[jetty-users] MappedByteBufferPool not pooling

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[jetty-users] MappedByteBufferPool not pooling
From: Jay C <dev.cyborg@xxxxxxxxx>
Date: Fri, 27 Dec 2013 00:23:41 -0800 (PST)
Delivered-to: jetty-users@xxxxxxxxxxx
List-archive: <https://dev.eclipse.org/mailman/private/jetty-users>
List-help: <mailto:jetty-users-request@eclipse.org?subject=help>
List-subscribe: <https://dev.eclipse.org/mailman/listinfo/jetty-users>, <mailto:jetty-users-request@eclipse.org?subject=subscribe>
List-unsubscribe: <https://dev.eclipse.org/mailman/options/jetty-users>, <mailto:jetty-users-request@eclipse.org?subject=unsubscribe>
Hello everyone,

I am new here. I recently started using Jetty 9.1 to serve a webapp making
intensive use of websockets with a certain success. Unfortunately the app
runs out of memory eventually.

I have read  Emmeran's recent post
<http://jetty.4.x6.nabble.com/jetty-users-DirectByteBuffer-native-memory-problems-OOM-with-Jetty-9-0-9-1-tp4961615.html>  
about how "big writes" might lead to OOM issues. 

In my situation it is rather the opposite as my app makes many "small
writes".  Typically, the webapp receives a stream of messages from a backend
app via websocket and broadcasts them to the browsers via websocket as well.   
The browsers never send any data on the websocket.

This is what happens after ~4 hours, there are 7M instances of
DirectByteBuffers:

-Environment:
Jetty v9.1.0.v20131115
JDK 1.7.0_45
Ubuntu 12.10 x64

-Memory profile:
java.nio.DirectByteBuffer                                  465371136  
465,371,136 B (55.2%)   7,271,424 (38.9%)
sun.misc.Cleaner                                             145428440  
145,428,440 B (17.2%)   3,635,711 (19.5%)
java.nio.DirectByteBuffer$Deallocator                  116342688  
116,342,688 B (13.8%)   3,635,709 (19.5%)
java.util.concurrent.ConcurrentLinkedQueue$Node   87265560    87,265,560 B
(10.3%)   3,636,065 (19.5%)

Caused by: java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:658)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
        at
org.eclipse.jetty.util.BufferUtil.allocateDirect(BufferUtil.java:122)
        at
org.eclipse.jetty.io.MappedByteBufferPool.acquire(MappedByteBufferPool.java:59)
        at
org.eclipse.jetty.websocket.common.extensions.compress.DeflateFrameExtension.outgoingFrame(DeflateFrameExtension.java:173)
        at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.outgoingFrame(ExtensionStack.java:235)
        at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendFrame(WebSocketRemoteEndpoint.java:168)
        at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendAsyncFrame(WebSocketRemoteEndpoint.java:107)
        at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendStringByFuture(WebSocketRemoteEndpoint.java:354)

Switching between blocking and non-blocking writes won't change the outcome.

Each message is ~70 bytes before deflate and ~10 bytes after deflate.
Typically there are ~200 msg/s.


I observed that for each message:

- AbstractWebSocketConnection acquires a buffer from ArrayByteBufferPool to
read the incoming message

qtp1477190004-58@6990, prio=5, in group 'main', status: 'RUNNING'
	  at
org.eclipse.jetty.io.*ArrayByteBufferPool.acquire*(ArrayByteBufferPool.java:64)
	  at
org.eclipse.jetty.websocket.common.Parser.parsePayload(Parser.java:563)
	  at org.eclipse.jetty.websocket.common.Parser.parseFrame(Parser.java:525)
	  at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:240)
	  at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.read(AbstractWebSocketConnection.java:596)
	  at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:503)
	  at
org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505)
	  at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
	  at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
	  at java.lang.Thread.run(Thread.java:744)

- DeflateFrameExtension acquires and allocates a new direct buffer from
MappedByteBufferPool to write the outgoing message

qtp1477190004-58@6990, prio=5, in group 'main', status: 'RUNNING'
	  at org.eclipse.jetty.util.BufferUtil.allocateDirect(BufferUtil.java:122)
	  at
org.eclipse.jetty.io.*MappedByteBufferPool.acquire*(MappedByteBufferPool.java:59)
	  at
org.eclipse.jetty.websocket.common.extensions.compress.DeflateFrameExtension.outgoingFrame(DeflateFrameExtension.java:173)
	  at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.outgoingFrame(ExtensionStack.java:235)
	  at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendFrame(WebSocketRemoteEndpoint.java:168)
	  at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.blockingWrite(WebSocketRemoteEndpoint.java:88)
	  at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendString(WebSocketRemoteEndpoint.java:331)
/*  handle incoming, crunch, send */
	  at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source:-1)
	  at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	  at java.lang.reflect.Method.invoke(Method.java:606)
	  at
org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:71)
	  at
org.eclipse.jetty.websocket.common.events.annotated.OptionalSessionCallableMethod.call(OptionalSessionCallableMethod.java:72)
	  at
org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onBinaryMessage(JettyAnnotatedEventDriver.java:107)
	  at
org.eclipse.jetty.websocket.common.message.SimpleBinaryMessage.messageComplete(SimpleBinaryMessage.java:68)
	  at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:62)
	  at
org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onBinaryFrame(JettyAnnotatedEventDriver.java:99)
	  at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:154)
	  at
org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:300)
	  at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202)
	  at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:206)
	  at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:243)
	  at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.read(AbstractWebSocketConnection.java:596)
	  at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:503)
	  at
org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505)
	  at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
	  at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
	  at java.lang.Thread.run(Thread.java:744)

- AbstractWebSocketConnection releases a buffer from ArrayByteBufferPool
after.. a successful write? 

qtp1477190004-58@6990, prio=5, in group 'main', status: 'RUNNING'
	  at
org.eclipse.jetty.io.*ArrayByteBufferPool.release*(ArrayByteBufferPool.java:78)
	  at
org.eclipse.jetty.websocket.common.io.WriteBytesProvider.releasePayloadBuffer(WriteBytesProvider.java:360)
	  at
org.eclipse.jetty.websocket.common.io.WriteBytesProvider$FrameEntry.freeBuffers(WriteBytesProvider.java:105)
	  at
org.eclipse.jetty.websocket.common.io.WriteBytesProvider$FrameEntry.notifySucceeded(WriteBytesProvider.java:83)
	  at
org.eclipse.jetty.websocket.common.io.WriteBytesProvider.succeeded(WriteBytesProvider.java:407)
	  at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:357)
	  at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:125)
	  at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.write(AbstractWebSocketConnection.java:688)
	  at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.flush(AbstractWebSocketConnection.java:359)
	  at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:569)
	  at
org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextOutgoingFrame(AbstractExtension.java:168)
	  at
org.eclipse.jetty.websocket.common.extensions.compress.DeflateFrameExtension.outgoingFrame(DeflateFrameExtension.java:213)
	  at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.outgoingFrame(ExtensionStack.java:235)
	  at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendFrame(WebSocketRemoteEndpoint.java:168)
	  at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.blockingWrite(WebSocketRemoteEndpoint.java:88)
	  at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendString(WebSocketRemoteEndpoint.java:331)
/*  handle incoming, crunch, send */
	  at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source:-1)
	  at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	  at java.lang.reflect.Method.invoke(Method.java:606)
	  at
org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:71)
	  at
org.eclipse.jetty.websocket.common.events.annotated.OptionalSessionCallableMethod.call(OptionalSessionCallableMethod.java:72)
	  at
org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onBinaryMessage(JettyAnnotatedEventDriver.java:107)
	  at
org.eclipse.jetty.websocket.common.message.SimpleBinaryMessage.messageComplete(SimpleBinaryMessage.java:68)
	  at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:62)
	  at
org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onBinaryFrame(JettyAnnotatedEventDriver.java:99)
	  at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:154)
	  at
org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:300)
	  at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202)
	  at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:206)
	  at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:243)
	  at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.read(AbstractWebSocketConnection.java:596)
	  at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:503)
	  at
org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505)
	  at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
	  at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
	  at java.lang.Thread.run(Thread.java:744)


At this point I am starting to wonder if that is the intended behaviour ?

- the MappedByteBufferPool buffers are always empty

- MappedByteBufferPool.release() is never called, but
MappedByteBufferPool.acquire() is called for each
WebSocketRemoteEndPoint.sendString() 

- both Mapped and Array ByteBufferPool implementations are being used
throughout different parts of Jetty 


Any idea regarding this issue anyone?

Thanks.

Jay



--
View this message in context: http://jetty.4.x6.nabble.com/MappedByteBufferPool-not-pooling-tp4961821.html
Sent from the Jetty User mailing list archive at Nabble.com.
Follow-Ups:
- Re: [jetty-users] MappedByteBufferPool not pooling
  - From: Joakim Erdfelt
- Re: [jetty-users] MappedByteBufferPool not pooling
  - From: Stefan Magnus Landrø
Prev by Date: [jetty-users] activemq-web and continuations
Next by Date: Re: [jetty-users] MappedByteBufferPool not pooling
Previous by thread: [jetty-users] activemq-web and continuations
Next by thread: Re: [jetty-users] MappedByteBufferPool not pooling
Index(es):
- Date
- Thread
Breadcrumbs
