Re: [jetty-users] over 200K open files FIFO pipe

[Luke B <lukenbutters@xxxxxxxxx>]

Hi,

This is with curl, making a GET request. It feels unlikely to me that curl is not sending all the data it should.

Curl version:

curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.36 zlib/1.2.7 libidn/1.28 libssh2/1.4.3

Is it possible to find out what these classes are that are holding this open?

I doub't it is the timeout because a timeout would maintain a constant amount of open pipes, instead I see it go up to multiple tens of thousands and dropping back to only a few thousand. I am sending GET requests single threaded one after the other and it seems to be a steady rate.

-Luke

On Sun, Sep 8, 2019 at 9:43 PM Simone Bordet <sbordet@xxxxxxxxxxx> wrote:

Hi,

On Sun, Sep 8, 2019 at 10:09 AM Luke B <lukenbutters@xxxxxxxxx> wrote:
>
> Hi,
>
> I am running jetty server "9.4.19.v20190610".
> I have no .wars loaded and have made a bunch of http 1.0 requests to the jetty web server to a URL which does not exist resulting in a 404 from the jetty web server.
>
> What I noticed is that after some time the web server stopped responding, I was unable to run jmap to get a heap dump I instead got the message too many open files.
>
> Running lsof on the process gave me lines like:
> Service    5309 5347    root  515r     FIFO                0,9       0t0  172660316 pipe
> Service    5309 5347    root  516w     FIFO                0,9       0t0  172660316 pipe
> Service    5309 5347    root  517r     FIFO                0,9       0t0  172660317 pipe
> Service    5309 5347    root  518w     FIFO                0,9       0t0  172660317 pipe
> Service    5309 5347    root  519r     FIFO                0,9       0t0  172661057 pipe
> Service    5309 5347    root  520w     FIFO                0,9       0t0  172661057 pipe
>
>  cat openfiles.lsof.txt | grep 5309 | grep FIFO | wc
>  211947 2115471 18863283
>
> is that expected?

HTTP 1.0 it's a request per connection:
* client opens the connection, send the request;
* server send response + content then shutdown output / close;
* client closes.

Is it possible that the client declares to send some content but it sends less?
That would leave the server with the open connection waiting for the
client to send the content.
The server will idle timeout the connection after (by default) 30
seconds anyway, but in that window of time you may see the number of
connections increasing.

Does it happen with HTTP/1.1?

> I restarted jetty and saw the number of those open files go up and down, like they got created and later gc'd.

It's probably the idle timeout kicking in, rather than the GC.

> p.s. I found this while trying to investigate a memory leak in conscrypt and jetty.

We'd love to hear what you find for this Conscrypt issue.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users