Re: [jetty-dev] jetty and 'hot-deploy', not freeing diskspace?

[Brian Reichert <reichert@xxxxxxxxxxx>]

On Wed, Aug 24, 2016 at 01:17:01PM -0400, Brian Reichert wrote:
> On Tue, Aug 23, 2016 at 05:11:24PM -0500, Jesse McConnell wrote:
> > Hi Brian,
> > 
> > Did you take a peek through these? It is typically the reason we see this
> > sort of behavior.
> > 
> > https://www.eclipse.org/jetty/documentation/current/preventing-memory-leaks.html
> 
> Will do; thanks for the pointer!

That was a quick read. :)

The pattern we see seems specific to the behavior of hot-deploy.

This is easily reproducible with stock jetty code:

In one window, spin up the demo-base app, with hot-deploy set:

  # cd /usr/jetty-distribution-9.3.8.v20160314/demo-base/
  # java -jar ../start.jar jetty.deploy.scanInterval=1

In another window, isolate the process ID, and show the current
count of deleted files, and the usage of the /tmp partition.

  # lsof -i :8080
  COMMAND  PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
  java    1463 root   89u  IPv6 5047191      0t0  TCP *:webcache (LISTEN)
  
  # lsof -nP +L1 | grep '(deleted)' | grep 1463 | wc -l
  0
  # df /tmp
  Filesystem     1K-blocks   Used Available Use% Mounted on
  /dev/sda2        2519792 470052   1921740  20% /
  # lsof -nP +L1 | grep '(deleted)' | grep 1463 | wc -l
  0

And periodically trigger hot-deploy, and observe the presence of
deleted files that the PID is still holding a reference to, and the
increase of utilization of the /tmp partition:

  # touch webapps/test.xml
  # lsof -nP +L1 | grep '(deleted)' | grep 1463 | wc -l
  7
  # df /tmp
  Filesystem     1K-blocks   Used Available Use% Mounted on
  /dev/sda2        2519792 470908   1920884  20% /

  # touch webapps/test.xml
  # lsof -nP +L1 | grep '(deleted)' | grep 1463 | wc -l
  14
  # df /tmp
  Filesystem     1K-blocks   Used Available Use% Mounted on
  /dev/sda2        2519792 471764   1920028  20% /

  # touch webapps/test.xml
  # lsof -nP +L1 | grep '(deleted)' | grep 1463 | wc -l
  21
  # df /tmp
  Filesystem     1K-blocks   Used Available Use% Mounted on
  /dev/sda2        2519792 472620   1919172  20% /
  
These count of deleted-but-referenced file keeps increasing, and
more of /tmp keeps getting consumed.

All of the deleted files are of the pattern:

  /tmp/jetty-0.0.0.0-8080-test.war-_test-any-XXXXXXXXXXXXXXXX.dir/...

Does anyone else see this behavior?

Again, details of our environment:

Details of our environment:

  jetty-distribution-9.3.8.v20160314

  OpenJDK Runtime Environment (build 1.8.0_71-b15)

  CentOS release 6.7 (Final)

  # uname -r
  2.6.32-573.18.1.el6.x86_64


-- 
Brian Reichert				<reichert@xxxxxxxxxxx>
BSD admin/developer at large