Hi Dave,
I see nothing wrong with this heap dump. It only shows a 200MB
memory usage, with nothing outstanding and nothing relating to the
2GB memory usage of the callstack analysis you were mentioning.
Maybe you can manually acquire a heap dump when memory usage is
high, using Memory Analyzer. You could trigger a garbage
collection just before to make sure it's actual memory and not
lingering unreachable objects.
Cheers,
Geneviève
On 2019-01-17 3:46 p.m., David Wootton
wrote:
Genevieve
I uploaded the file again
Download link https://framadrop.org/r/4_o3TANYNE#wA/uZVxcaNjzpwEh6TCLahcs+nzX4kkPGyi+eYkKf7o=
File delete link https://framadrop.org/d/4_o3TANYNE/6KC4F56ytyJPEB63i0aJIP2ioOcelt_M
Dave
Genevieve Bastien ---01/17/2019
03:34:02 PM---Hi Dave, Thanks, but the second link was to
delete the file... I clicked on it, the file was deleted
From: Genevieve
Bastien <gbastien+lttng@xxxxxxxxxxxx>
To: tracecompass-dev@xxxxxxxxxxx
Date: 01/17/2019
03:34 PM
Subject: Re:
[tracecompass-dev] Fw: Heap storage consumption for traces
Sent by: tracecompass-dev-bounces@xxxxxxxxxxx
Hi Dave,
Thanks, but the second link was to delete the file... I clicked
on it, the file was deleted, I don't have it... Can you upload
it again please? Sorry about that.
Geneviève
On 2019-01-17 2:37 p.m., David Wootton wrote:
Genevieve
I created a heap dump using memory sampling where I started
sampling at the point just before I started loading the
trace and stopped sampling after the trace was fully loaded
and messages in the Eclipse status bar were no longer
changing. I did not click the garbage collect icon in the
Eclipse toolbar so there's probably leftover objects in the
dump.
The original trace file was a bit more than 300MB. I
compressed it using gzip before uploading.
These are the links I got from the website https://framadrop.org/r/G96lAPv4xo#0cxhLW3Q9UQBZAeOQHy6YmwN85GQ4hPROpL1TDqAzlQ= to download and https://framadrop.org/d/G96lAPv4xo/KfH4omGHh8gvDLDhK0PlsutR8dFteLvJ to delete. The website says the file expires on
Jan 31.
Dave
"Geneviève Bastien" ---01/16/2019
03:21:41 PM---Hi Dave, Sorry to hear those improvements
didn't impact the problems you have. If you could share yo
From: "Geneviève Bastien" <gbastien@xxxxxxxxxxxx>
To: tracecompass-dev@xxxxxxxxxxx
Date: 01/16/2019 03:21 PM
Subject: Re: [tracecompass-dev] Fw:
Heap storage consumption for traces
Sent by: tracecompass-dev-bounces@xxxxxxxxxxx
Hi Dave,
Sorry to hear those improvements didn't impact the problems
you have. If you could share your heap dump with us, we
could take a look. You can use https://framadrop.org/
to share the file. If it's more than 200Mo, it will only be
valid for 24 hours, but we'll read our mails by then and
pick it up ;-)
Cheers,
Geneviève
On 2019-01-15 5:12 p.m., David Wootton wrote:
I built code using these instructions
and ran some crude side by side tests using the
Eclipse heap storage monitor in the status line. I
was puzzled that I didn't see any change in storage
utilization (around 2GB for 1.5 million events with
about 10 attributes per event) or time (around 1.5
minutes)
It turns out my analysis modules extend
CallStackAnalysis and the changes apply to
CallGraphAnalysis where the type hierarchies don't
intersect.
Dave
Matthew Khouzam ---01/09/2019
09:43:01 AM---Hi David, Sorry for the long delay,
the easiest way to make a trace compass baseline is
here.
From: Matthew Khouzam <matthew.khouzam@xxxxxxxxxxxx>
To: tracecompass developer
discussions <tracecompass-dev@xxxxxxxxxxx>
Date: 01/09/2019 09:43 AM
Subject: Re:
[tracecompass-dev] Fw: Heap storage consumption for
traces
Sent by: tracecompass-dev-bounces@xxxxxxxxxxx
Hi David,
Sorry for the long delay, the easiest way to make a
trace compass baseline is here.
First check out this page.
https://projects.eclipse.org/projects/tools.tracecompass/developer
Eclipse Trace Compass |
projects.eclipse.org
projects.eclipse.org
You can use the code from these repositories
to experiment, test, build, create patches,
issue pull requests, etc. This project uses
Gerrit Code Review; please see contributing
via Gerrit. |
Then checkout the patch in your
git repo like this.
$ git fetch ssh://yourname@xxxxxxxxxxxxxxx:29418/tracecompass/org.eclipse.tracecompass refs/changes/63/134363/4 &&
git checkout FETCH_HEAD
then run a maven build (mvn clean install
-Pdeploy-update-site) more info is here.
https://git.eclipse.org/c/tracecompass/org.eclipse.tracecompass.git/tree/README.md
You should then have a local update site you can
use.
From: tracecompass-dev-bounces@xxxxxxxxxxx <tracecompass-dev-bounces@xxxxxxxxxxx> on behalf of David Wootton <dwootton@xxxxxxxxxx>
Sent: Monday,
January 7, 2019 3:22:55 PM
To: tracecompass
developer discussions
Subject: Re:
[tracecompass-dev] Fw: Heap storage consumption for
traces
Matthew
I don't have an easy way to build a workspace with
pth Trace Compass and my plugin code. At the moment,
trying this is not urgent, so I'll wait until this
code gets into a repository I can work with.
Dave
Matthew Khouzam ---01/07/2019
11:21:17 AM---The patch set is not yet fully merged,
you can get a sneaky preview here. https://urldefense.proofpo
From: Matthew
Khouzam <matthew.khouzam@xxxxxxxxxxxx>
To: tracecompass
developer discussions <tracecompass-dev@xxxxxxxxxxx>
Date: 01/07/2019
11:21 AM
Subject: Re:
[tracecompass-dev] Fw: Heap storage consumption for
traces
Sent by: tracecompass-dev-bounces@xxxxxxxxxxx
The patch set is not yet fully merged, you can get a
sneaky preview here. https://git.eclipse.org/r/#/c/134363/
From: tracecompass-dev-bounces@xxxxxxxxxxx <tracecompass-dev-bounces@xxxxxxxxxxx> on behalf of David Wootton <dwootton@xxxxxxxxxx>
Sent: Monday,
January 7, 2019 10:42:48 AM
To: tracecompass
developer discussions
Subject: Re:
[tracecompass-dev] Fw: Heap storage consumption for
traces
Matthew.
That's good news. Will this be in the Simrel 2019-03
release? Will there be a Trace Compass repository
site I can point to at some point in time to give
this a quick look?
Thanks to Genevieve for the optimization work.
Dave
Matthew Khouzam ---01/07/2019
10:20:52 AM---Hey, Dave, Happy Holidays! Hope Santa
was good for you, she was for me, her name is
Genevieve and sh
From: Matthew
Khouzam <matthew.khouzam@xxxxxxxxxxxx>
To: tracecompass
developer discussions <tracecompass-dev@xxxxxxxxxxx>
Date: 01/07/2019
10:20 AM
Subject: Re:
[tracecompass-dev] Fw: Heap storage consumption for
traces
Sent by: tracecompass-dev-bounces@xxxxxxxxxxx
Hey, Dave,
Happy Holidays! Hope Santa was good for you, she was
for me, her name is Genevieve and she really
optimized the callgraph over the vacations. I think
you will be pleasantly surprised with the results.
😊
BR
Matthew
From: tracecompass-dev-bounces@xxxxxxxxxxx <tracecompass-dev-bounces@xxxxxxxxxxx> on behalf of David Wootton <dwootton@xxxxxxxxxx>
Sent: Thursday,
January 3, 2019 7:20:16 AM
To: tracecompass
developer discussions
Subject:
[tracecompass-dev] Fw: Heap storage consumption for
traces
Matthew
I hope you had a good holiday. I'm following up on
this now that I am back at work. Do you need
anything from me at this point, and how can I get it
to you?
Dave
----- Forwarded by David
Wootton/Poughkeepsie/Contr/IBM on 01/03/2019 07:18
AM -----
From: David
Wootton/Poughkeepsie/Contr/IBM
To: tracecompass
developer discussions <tracecompass-dev@xxxxxxxxxxx>
Date: 12/05/2018
02:28 PM
Subject: Re:
[tracecompass-dev] Heap storage consumption for
traces
Matthew
I think I can send you my heap dump, but I need to
check. It's about 200MB and I don't have any public
site I can upload it to. How do I get it to you?
I'm not really familiar with VisualVM so I'm not
sure how I check what owns an object.
I am a little surprised that I seem to only have 52
TmfStateValue objects when I have 4000
ITmfEventField objects. Maybe my placement for the
printf call is wrong and most of the objects have
been released, I placed it in my class overriding
the call stack view, in the traceOpened method. But
I am using a hash table to keep only a single
instance of a value.
If I understand, in TraceCompass 4.1, at the point
where I create a TmfStateValue to wrap an object
(String, Long, Integer, etc), I can now just use
that object itself in the call to the state builder
pushAttribute call. Is that correct?
Happy holidays
Dave
Matthew Khouzam ---12/05/2018
10:30:53 AM---Hi Dave, these are some pretty cool
findings. Now one thing to keep in mind, we have
several caches,
From: Matthew
Khouzam <matthew.khouzam@xxxxxxxxxxxx>
To: tracecompass
developer discussions <tracecompass-dev@xxxxxxxxxxx>
Date: 12/05/2018
10:30 AM
Subject: Re:
[tracecompass-dev] Heap storage consumption for
traces
Sent by: tracecompass-dev-bounces@xxxxxxxxxxx
Hi Dave,
these are some pretty cool findings. Now one thing
to keep in mind, we have several caches, so they do
trigger visualVM and rightly so. Can you look up who
is the owner of the mapEntries? Maybe we have a leak
with a manager or something.
Also, please note, tmfstatevalues are on the road to
deprecation. The state system now handles objects,
so you can save a few megabytes of useless pointers
by just using these objects. 😊
If you are allowed to share your heapdump, we can
look at that and try to optimize a bit here.
There is finally a possibility, but not a large one.
Does your EventField have a backreference to the
event, which contains a map of its fields? That
would be a prime candidate for where things can go
wrong.
BR,
Matthew. (Is it too soon to wish a happy holiday
season?)
From: tracecompass-dev-bounces@xxxxxxxxxxx <tracecompass-dev-bounces@xxxxxxxxxxx> on behalf of David Wootton <dwootton@xxxxxxxxxx>
Sent: Wednesday,
December 5, 2018 8:58:57 AM
To: tracecompass
developer discussions
Subject:
[tracecompass-dev] Heap storage consumption for
traces
A while back, Matthew and I discussed heap storage
utilization by traces. My traces have a number of
attributes associated with event, with the number of
attributes varying by type of trace. I create
ITmfEventField and TmfStateValue objects for each
attribute.
I have a trace with about 88,000 events. In that
trace, I ended up with close to 800,000
ITmfEventField and 500,000 TmfStateValue objects.
I created a hash table for ITmfEventField objects
that contained a single instance for each unique
field name and value pair. In this trace, I ended up
with about 4,000 unique ITmfEventField objects.
I created a hash table for the TmfStateValue objects
and end up with 52 unique objects.
I ran my code with VisialVM, and I noticed that the
top class for heap usage was
org.eclipse.tracecompass.internal.statesystem.core.backend.historytree.HTInterval.
From one run to the next the total number of objects
varies somewhere between 1.5 million and 2.5
million. If I click the Perform-GC button in
VisualVM, the count drops to around 700,000, fairly
consistently.
I also see a large number of instances of
com.google.com.collect.ImmutableMapEntry (355,000),
com.google.com.collect.ImmutableMapEntry[] (178,000)
classes and
com.google.com.collect.ImmutableMapEntry$NonTerminalImmutableMapEntry
(355,000)
Looking at the reference pane in VisualVM for a few
(< 10) of each of these classes, it looks like
the ImmutableMapEntry and
ImmutableMapEntry$nonTerminalImmutableMap objects
might be related to StateItem objects and the
ImmutableMapEntry[] objects might be related to
ITmfEventField objects.
I'm wondering if it is possible to use a hash table
to maintain single copies of unique instances of
those objects or whether the data in them guarantees
each instance is unique. If it is possible, then is
the performance tradeoff for ushing a hash table
worth it?
Dave_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your
password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your
password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your
password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your
password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your
password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your
password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password,
or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
|