Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jetty-users] Jetty leaves open sockets
Your QueuedThreadPool is way too small.

our advice for a minimum number is calculated based on your environment.

minimum = (number of cpu cores * (acceptors + selectors)) + 8

With your setup now, you are starving jetty's ability to process the networking.
When a task to cleanup the networking channel was offered to the threadpool it likely failed because the threadpool was exhausted.

--
Joakim Erdfelt <joakim@xxxxxxxxxxx>
Expert advice, services and support from from the Jetty & CometD experts


On Thu, Apr 17, 2014 at 5:27 AM, <bleshik@xxxxxxxxxxxxxx> wrote:
Hello!
We have a web service (Jersey based) on Jetty 9.1.3.v2014022 which somehow leaves sockets open. After a while we realized that problem occurs only in our production environment where we have a monitoring daemon calling the service with a simple ping request every 1 minute and a balancer doing the same thing every 6 seconds. The interesting part is the balancer ends its requests with RST (not FIN/ACK). No one else uses the service yet.
The service is pretty simple and is going to be called once a day for integration purposes so the configuration is fairly modest: 1 acceptor, 2 selectors and 2 threads for request handlers. Omitting irrelevant code it's roughly going like this:
final QueuedThreadPool threadPool = new QueuedThreadPool(5);
final Server server = new Server(threadPool);
final Connector[] connectors = new Connector[1];
final ServerConnector serverConnector = new ServerConnector(server, 1, 2);
serverConnector.setSoLingerTime(0);
serverConnector.setPort(35980);
connectors[0] = serverConnector;
server.setConnectors(connectors);
final ServletContextHandler context = new ServletContextHandler(ServletContextHandler.NO_SESSIONS);
context.setContextPath("/");
ServletHolder restApiHolder = ...;
context.addServlet(restApiHolder, "/*");
server.setHandler(context);
server.start();
The problem occurs pretty rarely. Here what we found out so far:
1) The service stops closing sockets immediately after two consequent exceptions:
[2014-04-14 12:29:50,552] DEBUG [qtp782189620-16 - /ping] write exception
org.eclipse.jetty.io.EofException
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:189)
        at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:335)
        at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:125)
        at org.eclipse.jetty.server.HttpConnection$CommitCallback.process(HttpConnection.java:555)
        at org.eclipse.jetty.util.IteratingCallback.processIterations(IteratingCallback.java:166)
        at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:126)
        at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:296)
        at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:715)
        at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:751)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:130)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:124)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:356)
        at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:154)
        at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:307)
        at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:261)
        at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:276)
        at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:835)
        at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:411)
        at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:645)
        at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:381)
        at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:371)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:262)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:318)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:236)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1010)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:382)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:345)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:220)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:711)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1644)
        at ru.yandex.bayan2.integration.servlet.LoggingFilter.doFilter(LoggingFilter.java:27)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1615)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1112)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:479)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1046)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:462)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:281)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
        at org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:744) 
Caused by: java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
        at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
        at sun.nio.ch.IOUtil.write(IOUtil.java:148)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:524)
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:169)
        ... 50 more
2) When it starts doing so, we get lots of such exceptions:
[2014-04-14 12:41:55,524] DEBUG [Scheduler-340008923] SelectChannelEndPoint@f67ae6c{/10.0.0.1:60468<->35980,Open,in,out,-,-,30000,HttpConnection}{io=0,kio=0,kro=1} idle timeout expired
[2014-04-14 12:41:55,524] DEBUG [Scheduler-340008923] ignored: WriteFlusher@7fdc861b{IDLE} java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
[2014-04-14 12:41:55,529] DEBUG [Scheduler-340008923] SelectChannelEndPoint@69f1af68{/10.0.0.1:48730<->35980,Open,in,out,-,-,30000,HttpConnection}{io=0,kio=0,kro=1} idle timeout check, elapsed: 30000 ms, remaining: 0 ms
but sockets never get closed.
3) org.eclipse.jetty.io.SelectorManager reports a growing number of selected events:
...
LOG.debug("Selector loop woken up from select, {}/{} selected", selected, _selector.keys().size());
...
4) Sockets left in CLOSE_WAIT state.
As a result we have a growing number of sockets and finally "Too many open files" exception.
So the questions are
1) Where can I dig further?
2) Why AbstractEndPoint#onIdleExpired(TimeoutException) doesn't simply close itself. Instead it checks some conditions before doing that:
    @Override
    protected void onIdleExpired(TimeoutException timeout)
    {
        boolean output_shutdown=isOutputShutdown();
        boolean input_shutdown=isInputShutdown();
        _fillInterest.onFail(timeout);
        _writeFlusher.onFail(timeout);
        if (isOpen() && output_shutdown || input_shutdown)
            close();
    }
3) Can this help us?
        ServerConnector connector = new ServerConnector(...) {
            @Override
            protected SelectChannelEndPoint newEndPoint(SocketChannel channel, SelectorManager.ManagedSelector selectSet, SelectionKey key) throws IOException {
                return new SelectChannelEndPoint(channel, selectSet, key, getScheduler(), getIdleTimeout()) {
                    @Override
                    protected void onIdleExpired(TimeoutException timeout) {
                        super.onIdleExpired(timeout);
                        close();
                    }
                };
            }
        };
Thanks.

_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/jetty-users


Back to the top