| [jetty-users] Jetty Deadlock between acceptor thread and connection handling thread? |
|
We recently ran into a bit of server trouble (requests hanging) and
upon taking a thread dump using jstack we found a deadlock at the
bottom. Here is the relevant application code (inside of WildfireServiceController.wildfireService) which is the deepest in the second thread's stack: byte[] bytes = info.getContent(gzip); response.setHeader("Cache-Control", "must-revalidate"); response.setDateHeader("Last-Modified", info.getLastModified().getTime()); response.setCharacterEncoding( "UTF-8" ); response.setContentType( "text/_javascript_" ); response.setContentLength( bytes.length ); if ( gzip ) { response.setHeader("Content-Encoding", "gzip"); } OutputStream os = null; try { os = response.getOutputStream(); os.write(bytes); os.flush(); } catch( IOException e ) { // This is usually caused when someone disconnects their browser prior to the file being streamed log.debug("IOException while streaming file.", e); } return null; One thing that immediately caught my eye is we're not explicitly calling os.close() here in a finally block - we assumed this wasn't necessary because we figured jetty would be doing that for us once the response was committed/written/flushed to the client. Is that a correct assumption? Deadlock Stacks: Found one Java-level deadlock: ============================= "1797400146@qtp-829571289-98 - Acceptor1 SelectChannelConnector@0.0.0.0:80": waiting to lock monitor 0x0000000042c99bc0 (object 0x00007f5043fead58, a org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint), which is held by "1818098997@qtp-829571289-94" "1818098997@qtp-829571289-94": waiting to lock monitor 0x00007f4fc04509e8 (object 0x00007f503a63cdf8, a org.mortbay.io.nio.SelectorManager$SelectSet), which is held by "1797400146@qtp-829571289-98 - Acceptor1 SelectChannelConnector@0.0.0.0:80" Java stack information for the threads listed above: =================================================== "1797400146@qtp-829571289-98 - Acceptor1 SelectChannelConnector@0.0.0.0:80": at org.mortbay.io.nio.SelectChannelEndPoint.updateKey(SelectChannelEndPoint.java:322) - waiting to lock <0x00007f5043fead58> (a org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint) at org.mortbay.io.nio.SelectChannelEndPoint.close(SelectChannelEndPoint.java:456) at org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint.close(SelectChannelConnector.java:362) at org.mortbay.io.nio.SelectChannelEndPoint.idleExpired(SelectChannelEndPoint.java:174) at org.mortbay.io.nio.SelectChannelEndPoint$IdleTask.expire(SelectChannelEndPoint.java:489) at org.mortbay.thread.Timeout.tick(Timeout.java:137) - locked <0x00007f503a63cdf8> (a org.mortbay.io.nio.SelectorManager$SelectSet) at org.mortbay.thread.Timeout.tick(Timeout.java:153) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:753) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:191) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) "1818098997@qtp-829571289-94": at org.mortbay.io.nio.SelectorManager$SelectSet.scheduleIdle(SelectorManager.java:784) - waiting to lock <0x00007f503a63cdf8> (a org.mortbay.io.nio.SelectorManager$SelectSet) at org.mortbay.io.nio.SelectChannelEndPoint.scheduleIdle(SelectChannelEndPoint.java:159) at org.mortbay.io.nio.SelectChannelEndPoint.blockWritable(SelectChannelEndPoint.java:293) - locked <0x00007f5043fead58> (a org.mortbay.jetty.nio.SelectChannelConnector$ConnectorEndPoint) at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:544) at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:571) at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1010) at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:650) at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:590) at wildfire.web.api.WildfireServiceController.wildfireService(WildfireServiceController.java:62) at sun.reflect.GeneratedMethodAccessor701.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:174) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:771) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:716) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166) at wildfire.web.interceptor.ContactFilter.doFilterInternal(ContactFilter.java:110) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at wildfire.web.interceptor.FacebookFilter.processFacebookEnabledRequest(FacebookFilter.java:130) at wildfire.web.interceptor.FacebookFilter.doFilterInternal(FacebookFilter.java:58) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at wildfire.web.interceptor.TrackingFilter.doFilterInternal(TrackingFilter.java:62) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at wildfire.web.interceptor.Log4jRequestInfoFilter.doFilterInternal(Log4jRequestInfoFilter.java:50) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.jsecurity.web.servlet.FilterChainWrapper.doFilter(FilterChainWrapper.java:52) at org.jsecurity.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:105) at org.jsecurity.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:135) at org.jsecurity.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:180) at org.jsecurity.web.servlet.FilterChainWrapper.doFilter(FilterChainWrapper.java:57) at org.jsecurity.web.servlet.JSecurityFilter.doFilterInternal(JSecurityFilter.java:382) at wildfire.util.jsecurity.CleanupJSecurityFilter.doFilterInternal(CleanupJSecurityFilter.java:20) at org.jsecurity.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:180) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at wildfire.web.interceptor.MaintenanceFilter.doFilterInternal(MaintenanceFilter.java:55) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at wildfire.web.filter.SslFilter.doFilterInternal(SslFilter.java:63) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at wildfire.web.interceptor.OrganizationFilter.doFilterInternal(OrganizationFilter.java:106) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:198) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at wildfire.web.filter.DomainFilter.doFilterInternal(DomainFilter.java:58) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at wildfire.web.filter.StaticResourcesFilter.doFilterInternal(StaticResourcesFilter.java:47) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at wildfire.web.interceptor.ProfilingFilter.doFilterInternal(ProfilingFilter.java:31) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at wildfire.web.filter.NoCacheFilter.doFilter(NoCacheFilter.java:49) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at wildfire.web.filter.PrivacyPolicyFilter.doFilterInternal(PrivacyPolicyFilter.java:20) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.terracotta.session.SessionFilter.doHttpFilter(SessionFilter.java:74) at com.terracotta.session.SessionFilter.doFilter(SessionFilter.java:58) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:926) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Found 1 deadlock. |