[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [mat-dev] java.io.FileNotFoundException: /mat-work/nikhil/MAT1.11/mat/../../sacbay-m-big77GB.inbound.index3096.log (Too many open files)
|
I would prefer not having a configuration option as that is an extra thing
to document and test and some users who need it might not even find out
about the option.
I think it would be better to aim for a maximum of 3500 to 4000 open
files, either by increasing the segment size or by capping the number of
segments.
The code seems to take the (number of identifiers + 1) / 500000 then find
the power of two just bigger than this, so for 1,168,946,725 identifiers
this could create 4096 segments.
I don't know why the number of segments is chosen to be a power of two.
The code including giant segments is explained here.
https://bugs.eclipse.org/bugs/show_bug.cgi?id=372548
It was updated for multithreaded with bug 277422 and a minor update made
inbound references to a class object go in ascending order.
So we need to understand if the number of segments needs to be a power of
two and what a good upper limit would be and how many other open files
there normally are.
Andrew Johnson
From: "Kevin Grigorenko" <kevin.grigorenko@xxxxxxxxxx>
To: "Memory Analyzer Dev list" <mat-dev@xxxxxxxxxxx>
Date: 14/07/2021 16:13
Subject: Re: [mat-dev] java.io.FileNotFoundException:
/mat-work/nikhil/MAT1.11/mat/../../sacbay-m-big77GB.inbound.index3096.log
(Too many open files)
Sent by: "mat-dev" <mat-dev-bounces@xxxxxxxxxxx>
ZjQcmQRYFpfptBannerEnd
Andrew, what do you think about a configurable setting for the 500,000
identifier split that Nikhil could increase?
--
Kevin Grigorenko
IBM App Platform SWAT
From: "Nikhil Dongre" <nikhil.vilas@xxxxxxxxx>
To: "Memory Analyzer Dev list" <mat-dev@xxxxxxxxxxx>
Date: 07/14/2021 06:10 AM
Subject: [EXTERNAL] Re: [mat-dev] java.io.FileNotFoundException:
/mat-work/nikhil/MAT1.11/mat/../../sacbay-m-big77GB.inbound.index3096.log
(Too many open files)
Sent by: "mat-dev" <mat-dev-bounces@xxxxxxxxxxx>
HI Kevin, Andrew, That means with the latest version we cannot process
Large Heap dumps? Regards Nikhil Dongre On Tue, Jul 13, 2021 at 11:03 PM
Kevin Grigorenko <kevin.grigorenko@xxxxxxxxxx> wrote: Hi Nikhil,
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
HI Kevin, Andrew,
That means with the latest version we cannot process Large Heap dumps?
Regards
Nikhil Dongre
On Tue, Jul 13, 2021 at 11:03 PM Kevin Grigorenko <
kevin.grigorenko@xxxxxxxxxx> wrote:
Hi Nikhil,
You might be able to set ulimits if you run a privileged container (which
generally isn't recommended) but otherwise I don't think you can. Example
reproducing your error:
$ docker run --rm fedora sh -c "whoami; ulimit -n; sleep 1; ulimit -n
10000000"
root
1048576
sh: line 1: ulimit: open files: cannot modify limit: Operation not
permitted
As Andrew mentioned, you would need to specify the ulimit when launching
the container; for example, for Docker: --ulimit nofile=90000
See
https://docs.docker.com/engine/reference/commandline/run/#set-ulimits-in-container---ulimit
--
Kevin Grigorenko
IBM App Platform SWAT
From: "Nikhil Dongre" <nikhil.vilas@xxxxxxxxx>
To: "Memory Analyzer Dev list" <mat-dev@xxxxxxxxxxx>
Date: 07/13/2021 10:00 AM
Subject: [EXTERNAL] Re: [mat-dev] java.io.FileNotFoundException:
/mat-work/nikhil/MAT1.11/mat/../../sacbay-m-big77GB.inbound.index3096.log
(Too many open files)
Sent by: "mat-dev" <mat-dev-bounces@xxxxxxxxxxx>
Hi Andrew,Kevin, I am not able to modify the ulimit beyond 4096. I tried
the below command and got an exception. [opc@ci-iad-mat-lab-01 ~]$ ulimit
-n 90000 -bash: ulimit: open files: cannot modify limit: Operation not
permitted ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
Hi Andrew,Kevin,
I am not able to modify the ulimit beyond 4096.
I tried the below command and got an exception.
[opc@ci-iad-mat-lab-01 ~]$ ulimit -n 90000
-bash: ulimit: open files: cannot modify limit: Operation not permitted
Here is ulimit command output
[opc@ci-iad-mat-lab-01 mat]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 239754
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Regards
Nikhil Dongre
On Mon, Jul 12, 2021 at 11:43 PM Nikhil Dongre <nikhil.vilas@xxxxxxxxx>
wrote:
Hi Andrew,
I think this is a very genuine scenario with 70GB + hprof files with MAT
1.11 as by default Linux os max support 4096 open files.
And in our case 4089 (sacbay-m-big77GB.inbound.index4089.log) files I can
see on my disk.
Once we started using MAT 1.11 it started falling for large hprof files
and in our scenario we are processing upto 100GB files with old MAT 1.8 it
was very very slow but rarely failed
And if we think of the above solution, is there any way to calculate the
on the basis of hprof file size? (Reason: upto 50GB hprof files we didn't
face any issue and we may need to process max 100GB file)
"It might cause performance or memory problems for other people if we
reduced the number and so increased the size of the open temporary index
files."
I agree with the above comment.
But is there any way to make this working for Large Heap Dumps.
Regards
Nikhil Dongre
On Fri, Jul 9, 2021 at 1:32 PM Andrew Johnson <andrew_johnson@xxxxxxxxxx>
wrote:
Nikhil,
The code for this is here:
https://git.eclipse.org/r/plugins/gitiles/mat/org.eclipse.mat/+/refs/tags/R_1.12.0/plugins/org.eclipse.mat.parser/src/org/eclipse/mat/parser/index/IndexWriter.java
I think the current code is a reasonable compromise - I think it has one
inbound.indexNNNN.log file for every 500,000 identifiers, so for the
maximum possible 2^31 identifiers it could allocate 4295 files.
There are suggestions on the Internet that you can change the number of
open files when starting a container:
https://docs.docker.com/engine/reference/commandline/run/#set-ulimits-in-container---ulimit
https://mtyurt.net/post/docker-how-to-increase-number-of-open-files-limit.html
E.g. $ docker run --ulimit nofile=1024:1024 --rm debian sh -c "ulimit -n"
It might cause performance or memory problems for other people if we
reduced the number and so increased the size of the open temporary index
files.
Regards,
Andrew Johnson
From: "Nikhil Dongre" <nikhil.vilas@xxxxxxxxx>
To: "Memory Analyzer Dev list" <mat-dev@xxxxxxxxxxx>
Date: 07/07/2021 17:56
Subject: [EXTERNAL] Re: [mat-dev] java.io.FileNotFoundException:
/mat-work/nikhil/MAT1.11/mat/../../sacbay-m-big77GB.inbound.index3096.log
(Too many open files)
Sent by: "mat-dev" <mat-dev-bounces@xxxxxxxxxxx>
Hi Andrew,
Thanks for quick work around.
But we have scenarios where we may not be able to change the limit as its
container based solution. Once Hprof files are processed and HTML reports
are generated the container will vanish and throughout we will not have
access to the container.
And another scenario is hprof file size it may vary from 1gb to 100gb so
fixing at OS level may not be the right solution.
Please suggest.
Regards
Nikhil Dongre
On Wed, Jul 7, 2021 at 5:09 PM Andrew Johnson <andrew_johnson@xxxxxxxxxx>
wrote:
Memory Analyzer does create a lot of files at the re-indexing stage as it
helps to build the inbound index. These are deleted once that stage is
over, so you don't normally notice them.
You need to increase the limit on the number of open files for your
process. Something like
ulimit -S -n 4096
would increase the soft limit for the process - if 4096 is too small for
MAT with this dump you will still get the error. There is a hard limit of
the number of open files:
ulimit -H -n
and an ordinary user can't increase the soft limit beyond the hard limit.
If you need more then you need root access or a system administrator to
increase the limits for your userid
The administrator might need to change something like the following.
/etc/security/limits.conf
* - nofile 16384
Once that is done you might need to log in again and run
ulimit -S -n 16384
Andrew Johnson
From: "Nikhil Dongre" <nikhil.vilas@xxxxxxxxx>
To: "Memory Analyzer Dev list" <mat-dev@xxxxxxxxxxx>
Date: 07/07/2021 06:51
Subject: Re: [mat-dev] java.io.FileNotFoundException:
/mat-work/nikhil/MAT1.11/mat/../../sacbay-m-big77GB.inbound.index3096.log
(Too many open files)
Sent by: "mat-dev" <mat-dev-bounces@xxxxxxxxxxx>
Here are some more details
Please find the command ulimit -a output below :
[opc@ci-iad-mat-lab-01 mat]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 239754
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[opc@ci-iad-mat-lab-01 mat]$
Hprof file size is : 77GB
MAT version : 1.11
And its failing after this step "Re-indexing outbound index"
[Task: Parsing
/mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.hprof
[
Subtask: Scanning
/mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.hprof
[....................
[INFO] Wrote threads call stacks to
/mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.threads
[....................
[INFO] Heap
/mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.hprof contains
1,168,946,725 objects
[....................
Subtask: Extracting objects from
/mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.hprof
[................................................................................Task:
Removing unreachable objects
[
Subtask: Searching for unreachable objects
[
Subtask: Marking reachable objects
[.
Subtask: Re-indexing objects
[.
[INFO] Removed 72,901,319 unreachable objects using 6,692,864,976 bytes
[..
Subtask: Re-indexing classes
[...
Subtask: Writing
/mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.idx.index
[....
Subtask: Writing
/mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.o2c.index
[.....
Subtask: Writing
/mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.a2s.index
[......
Subtask: Re-indexing outbound index
[......]
eclipse.buildId=unknown
java.version=1.8.0_292
java.vendor=Red Hat, Inc.
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Framework arguments: -application org.eclipse.mat.api.parse
-DhprofStrictnessWarning=true ../../sacbay-m-big77GB.hprof
org.eclipse.mat.api:suspects org.eclipse.mat.api:top_components
org.eclipse.mat.api:overview
Command-line arguments: -os linux -ws gtk -arch x86_64 -consolelog
-application org.eclipse.mat.api.parse -DhprofStrictnessWarning=true
../../sacbay-m-big77GB.hprof org.eclipse.mat.api:suspects
org.eclipse.mat.api:top_components org.eclipse.mat.api:overview
!ENTRY org.eclipse.osgi 4 0 2021-07-07 05:32:46.312
!MESSAGE Application error
!STACK 1
org.eclipse.mat.SnapshotException: Error opening heap dump
'sacbay-m-big77GB.hprof'. Check the error log for further details.
at
org.eclipse.mat.parser.internal.SnapshotFactoryImpl.parse(SnapshotFactoryImpl.java:331)
at
org.eclipse.mat.parser.internal.SnapshotFactoryImpl.openSnapshot(SnapshotFactoryImpl.java:167)
at
org.eclipse.mat.snapshot.SnapshotFactory.openSnapshot(SnapshotFactory.java:147)
at
org.eclipse.mat.internal.apps.ParseSnapshotApp.parse(ParseSnapshotApp.java:138)
at
org.eclipse.mat.internal.apps.ParseSnapshotApp.start(ParseSnapshotApp.java:110)
at
org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
at
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:137)
at
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:107)
at
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)
at
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:657)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:594)
at org.eclipse.equinox.launcher.Main.run(Main.java:1447)
at org.eclipse.equinox.launcher.Main.main(Main.java:1420)
Caused by: java.io.FileNotFoundException:
/mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.inbound.index3096.log
(Too many open files)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at
org.eclipse.mat.parser.index.IndexWriter$InboundWriter.log(IndexWriter.java:1237)
at
org.eclipse.mat.parser.internal.GarbageCleaner.clean(GarbageCleaner.java:360)
at
org.eclipse.mat.parser.internal.SnapshotFactoryImpl.parse(SnapshotFactoryImpl.java:282)
... 17 more
On Wed, Jul 7, 2021 at 9:03 AM Nikhil Dongre <nikhil.vilas@xxxxxxxxx>
wrote:
Hi Kevin,
Thanks for the quick response.
Please find the command output below :
[opc@ci-iad-mat-lab-01 mat]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 239754
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[opc@ci-iad-mat-lab-01 mat]$
Regards
Nikhil Dongre
On Wed, Jul 7, 2021 at 8:52 AM Kevin Grigorenko <
kevin.grigorenko@xxxxxxxxxx> wrote:
The error "Too many open files" may occur because your file ulimit is too
small, system-wide limits have been exceeded, or there is a bug in MAT
creating too many file descriptors (in decreasing order of probability).
Please start a terminal and send the output of the following command:
$ ulimit -a
--
Kevin Grigorenko
IBM App Platform SWAT
From: "Nikhil Dongre" <nikhil.vilas@xxxxxxxxx>
To: mat-dev@xxxxxxxxxxx
Date: 07/06/2021 06:01 PM
Subject: [EXTERNAL] [mat-dev] java.io.FileNotFoundException:
/mat-work/nikhil/MAT1.11/mat/../../sacbay-m-big77GB.inbound.index3096.log
(Too many open files)
Sent by: "mat-dev" <mat-dev-bounces@xxxxxxxxxxx>
Hi Team, When i tried to process process gz hprof file i got this
exception org.eclipse.mat.SnapshotException: Error opening heap dump
'sacbay-m-big77GB.hprof.gz'. Check the error log for further details. at
org.eclipse.mat.parser.internal.SnapshotFactoryImpl.parse(SnapshotFactoryImpl.java:331)
Hi Team,
When i tried to process process gz hprof file i got this exception
org.eclipse.mat.SnapshotException: Error opening heap dump
'sacbay-m-big77GB.hprof.gz'. Check the error log for further details.
at
org.eclipse.mat.parser.internal.SnapshotFactoryImpl.parse(SnapshotFactoryImpl.java:331)
at
org.eclipse.mat.parser.internal.SnapshotFactoryImpl.openSnapshot(SnapshotFactoryImpl.java:167)
at
org.eclipse.mat.snapshot.SnapshotFactory.openSnapshot(SnapshotFactory.java:147)
at
org.eclipse.mat.internal.apps.ParseSnapshotApp.parse(ParseSnapshotApp.java:138)
at
org.eclipse.mat.internal.apps.ParseSnapshotApp.start(ParseSnapshotApp.java:110)
at
org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
at
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:137)
at
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:107)
at
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)
at
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:657)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:594)
at org.eclipse.equinox.launcher.Main.run(Main.java:1447)
at org.eclipse.equinox.launcher.Main.main(Main.java:1420)
Caused by: java.io.FileNotFoundException:
/mat-work/nikhil/MAT1.11/mat/../../sacbay-m-big77GB.inbound.index3096.log
(Too many open files)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at
org.eclipse.mat.parser.index.IndexWriter$InboundWriter.log(IndexWriter.java:1237)
at
org.eclipse.mat.parser.internal.GarbageCleaner.clean(GarbageCleaner.java:360)
at
org.eclipse.mat.parser.internal.SnapshotFactoryImpl.parse(SnapshotFactoryImpl.java:282)
... 17 more
MemoryAnalyzer:
An error has occurred. See the log file
_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev
_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev
_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev
_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev
_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev
_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev
_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev
_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/mat-dev
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
