[jetty-dev] ReadPendingException in Jetty 9.4.8

[Varun Thacker <varun@xxxxxxxxxxx>]

Hi Everyone,

While running a production setup of Solr with Jetty 9.4.8 we ran into a scenario where the replica node hit's a ReadPendingException . The leader node keeps waiting for the request to complete but eventually times out with a socket timeout exception 

Here's the stack trace excerpt :

java.nio.channels.ReadPendingException: null
at org.eclipse.jetty.io.FillInterest.register(FillInterest.java:58) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121]
at org.eclipse.jetty.io.AbstractEndPoint.fillInterested(AbstractEndPoint.java:353) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121]

...

at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) ~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) ~[jetty-util-9.4.8.v20171121.jar:9.4.8.v20171121]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) ~[jetty-util-9.4.8.v20171121.jar:9.4.8.v20171121]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[jetty-util-9.4.8.v20171121.jar:9.4.8.v20171121]
at 

I found one existing bug https://github.com/eclipse/jetty.project/issues/1047 but this seems to have been addressed quite a while ago.

My current analysis was that we're seeing this because of a possible bug in EatWhatYouKill . Is this a known issue or something that needs more debug logs to investigate further?

The current strategy I am planning on taking is upgrading Jetty to 9.4.9+ and see if it fixes the problem. This class has gone through significant changes in Jetty 9.4.9 ( 

https://github.com/eclipse/jetty.project/commit/0cb4f5629dca082eec943b94ec8ef4ca0d5f1aa4#diff-ae450a12d4eca85a437bd5082f698f48 . 

 )