Re: [jetty-users] Jetty intermittently refusing requests (ConnectExcepti

[Simone Bordet <sbordet@xxxxxxxxxxx>]

Hi,

On Fri, Oct 20, 2017 at 2:52 PM, Sean Dawson <seandawson2015@xxxxxxxxx> wrote:
>
> Hello,
>
> We have a critical issue where our app running in Tomcat (8.x) attempts to
> make a request to our other app running in Jetty (9.2.7) with Java 8
> occasionally gets one or two ConnectExceptions which results in that app
> being marked unavailable - because our previous understanding was that we'd
> only receive that in cases where the app/server was physically down (or some
> other permanent issue like firewall, etc).
>
> We can't determine why this happens (in one of the two cases) or when it
> might happen (no pattern yet) and are at a loss on how to troubleshoot this
> further.  Is there logging for the OS or Jetty that might help here?  Or any
> other ideas why we might have this?

ConnectExceptions are thrown on the client when an attempt to connect
results in a RST packet sent by the server in response to an initial
SYN packet.
Jetty does not have control over this.

Do you have the full stack trace ?

You are running a very old Jetty version, current is 9.4.7.

> It's happened a few times at one customer site running Windows 2012 - but
> there it's always happened under extremely heavy load.  So my guess is that
> it's a thread or OS or port limit that's being encountered. (But we still
> can't say for sure yet.)
>
> The other case is on CentOS release 6.9 (Final) on a popular cloud server
> provider, and it's happened two separate times, and in neither was the
> server or applications under any kind of substantial load.  There's nothing
> in the (minimal) system logs or the app logs to indicate an issue.  Just a
> request from Tomcat that ends up getting a ConnectException and so we mark
> the server as unavailable.
>
> We use something along these lines for our Embedded Jetty server...
>
> WebAppContext context = new WebAppContext();
> QueuedThreadPool threadPool = new QueuedThreadPool(maxThreads); // usually
> 256
> Server server = new Server(threadPool);
> HttpConfiguration http_config = new HttpConfiguration();
> ServerConnector http = new ServerConnector(server, new
> HttpConnectionFactory(http_config));
> http.setPort(port);
> server.setConnectors(new Connector[]
> {
>    http
> });
> server.setHandler(context);
> server.start();
> server.join();

Nothing wrong here.

> The app works quite well most of the time but we're baffled under what
> circumstances Jetty would briefly refuse connections (when not under heavy
> load).  Nothing on the server indicates any issues at the time.
>
> # ulimit -a
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 1031937
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 8192

This is really small for a server. Try 512 K instead.

> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 10240
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 8192
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
>
> Memory of the server or app is not low at the time this occurs. And I've
> tried to reproduce in-house by forcing long gc cycles (or overloading the
> app) but we just end up getting timeouts rather than ConnectExceptions.  Any
> help appreciated.

Try to play with the parameter ServerConnector.acceptQueueSize (by
default 0, which means that the implementation default of 50 is being
used).
Try 200 or more and see if you still get these errors.

Please report back your findings.

-- 
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.