[jetty-users] Limits of HttpClient for uploading larger files (500+ MB)?

[Libor Jelinek <ljelinek@xxxxxxxxxxx>]

Hello!
I'm encountering OutOfMemoryError with Jetty HttpClient when sending larger files. Even 500 MB file with heap size is set to 2 GB causes OOME! It smells like a memory leak for me (and also for Eclipse Memory Analyzer Tool).

This simple snippet

HttpClient client = new HttpClient();
client.start();

ContentExchange ex = new ContentExchange(true) {
   @Override
   protected void onException(Throwable x) {

      throw new RuntimeException(x);
   }
};
ex.setMethod("PUT");
ex.setURL("http://localhost:8080/plexflow-api/push-begin?fileId=823");

ex.setFileForUpload(oneGbFile);

client.send(ex);

causes

java.lang.RuntimeException: java.lang.OutOfMemoryError

at ioutilsmemoryleak.Application$1.onException(Application.java:89) ~[IoUtilsMemoryLeak/:na]

at org.eclipse.jetty.client.HttpExchange$Listener.onException(HttpExchange.java:1103) ~[jetty-client-8.1.5.v20120716.jar:8.1.5.v20120716]

at org.eclipse.jetty.client.AsyncHttpConnection.handle(AsyncHttpConnection.java:159) ~[jetty-client-8.1.5.v20120716.jar:8.1.5.v20120716]

at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:622) ~[jetty-client-8.1.5.v20120716.jar:8.1.5.v20120716]

at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) [jetty-client-8.1.5.v20120716.jar:8.1.5.v20120716]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) [jetty-client-8.1.5.v20120716.jar:8.1.5.v20120716]

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) [jetty-client-8.1.5.v20120716.jar:8.1.5.v20120716]

at java.lang.Thread.run(Thread.java:722) [na:1.7.0_04]

Caused by: java.lang.OutOfMemoryError: null

at java.io.FileInputStream.readBytes(Native Method) ~[na:1.7.0_04]

at java.io.FileInputStream.read(FileInputStream.java:220) ~[na:1.7.0_04]

at org.eclipse.jetty.client.AbstractHttpConnection.commitRequest(AbstractHttpConnection.java:234) ~[jetty-client-8.1.5.v20120716.jar:8.1.5.v20120716]

at org.eclipse.jetty.client.AsyncHttpConnection.handle(AsyncHttpConnection.java:80) ~[jetty-client-8.1.5.v20120716.jar:8.1.5.v20120716]

... 5 common frames omitted

Have anybody else faced the same symptoms?

Thanks,

Libor

Attachment: leak in httpclient.png
Description: PNG image