Hi Dave,
Thanks, but the second link was to delete the file... I clicked
on it, the file was deleted, I don't have it... Can you upload it
again please? Sorry about that.
Geneviève
On 2019-01-17 2:37 p.m., David Wootton
wrote:
Genevieve
I created a heap dump using memory sampling where
I started sampling at the point just before I started loading
the trace and stopped sampling after the trace was fully
loaded and messages in the Eclipse status bar were no longer
changing. I did not click the garbage collect icon in the
Eclipse toolbar so there's probably leftover objects in the
dump.
The original trace file was a bit more than
300MB. I compressed it using gzip before uploading.
These are the links I got from the website https://framadrop.org/r/G96lAPv4xo#0cxhLW3Q9UQBZAeOQHy6YmwN85GQ4hPROpL1TDqAzlQ= to download and https://framadrop.org/d/G96lAPv4xo/KfH4omGHh8gvDLDhK0PlsutR8dFteLvJ to delete. The website says the file expires on Jan
31.
Dave
"Geneviève Bastien"
---01/16/2019 03:21:41 PM---Hi Dave, Sorry to hear those
improvements didn't impact the problems you have. If you could
share yo
From: "Geneviève
Bastien" <gbastien@xxxxxxxxxxxx>
To: tracecompass-dev@xxxxxxxxxxx
Date: 01/16/2019
03:21 PM
Subject: Re:
[tracecompass-dev] Fw: Heap storage consumption for traces
Sent by: tracecompass-dev-bounces@xxxxxxxxxxx
Hi Dave,
Sorry to hear those improvements didn't impact the problems you
have. If you could share your heap dump with us, we could take a
look. You can use https://framadrop.org/
to share the file. If it's more than 200Mo, it will only be
valid for 24 hours, but we'll read our mails by then and pick it
up ;-)
Cheers,
Geneviève
On 2019-01-15 5:12 p.m., David Wootton wrote:
I built code using these instructions and ran
some crude side by side tests using the Eclipse heap storage
monitor in the status line. I was puzzled that I didn't see
any change in storage utilization (around 2GB for 1.5
million events with about 10 attributes per event) or time
(around 1.5 minutes)
It turns out my analysis modules extend CallStackAnalysis
and the changes apply to CallGraphAnalysis where the type
hierarchies don't intersect.
Dave
Matthew Khouzam ---01/09/2019
09:43:01 AM---Hi David, Sorry for the long delay, the
easiest way to make a trace compass baseline is here.
From: Matthew Khouzam <matthew.khouzam@xxxxxxxxxxxx>
To: tracecompass developer
discussions <tracecompass-dev@xxxxxxxxxxx>
Date: 01/09/2019 09:43 AM
Subject: Re: [tracecompass-dev] Fw:
Heap storage consumption for traces
Sent by: tracecompass-dev-bounces@xxxxxxxxxxx
Hi David,
Sorry for the long delay, the easiest way to make a trace
compass baseline is here.
First check out this page.
https://projects.eclipse.org/projects/tools.tracecompass/developer
Eclipse
Trace Compass | projects.eclipse.org
projects.eclipse.org
You can use the code from these repositories to
experiment, test, build, create patches, issue pull
requests, etc. This project uses Gerrit Code Review;
please see contributing via Gerrit. |
Then checkout the patch in your git repo
like this.
$ git fetch ssh://yourname@xxxxxxxxxxxxxxx:29418/tracecompass/org.eclipse.tracecompass refs/changes/63/134363/4 && git
checkout FETCH_HEAD
then run a maven build (mvn clean install
-Pdeploy-update-site) more info is here.
https://git.eclipse.org/c/tracecompass/org.eclipse.tracecompass.git/tree/README.md
You should then have a local update site you can use.
From: tracecompass-dev-bounces@xxxxxxxxxxx <tracecompass-dev-bounces@xxxxxxxxxxx> on behalf of David Wootton <dwootton@xxxxxxxxxx>
Sent: Monday, January 7,
2019 3:22:55 PM
To: tracecompass developer
discussions
Subject: Re:
[tracecompass-dev] Fw: Heap storage consumption for traces
Matthew
I don't have an easy way to build a workspace with pth Trace
Compass and my plugin code. At the moment, trying this is
not urgent, so I'll wait until this code gets into a
repository I can work with.
Dave
Matthew Khouzam ---01/07/2019
11:21:17 AM---The patch set is not yet fully merged, you can
get a sneaky preview here. https://urldefense.proofpo
From: Matthew Khouzam
<matthew.khouzam@xxxxxxxxxxxx>
To: tracecompass
developer discussions <tracecompass-dev@xxxxxxxxxxx>
Date: 01/07/2019 11:21
AM
Subject: Re:
[tracecompass-dev] Fw: Heap storage consumption for traces
Sent by: tracecompass-dev-bounces@xxxxxxxxxxx
The patch set is not yet fully merged, you can get a sneaky
preview here. https://git.eclipse.org/r/#/c/134363/
From: tracecompass-dev-bounces@xxxxxxxxxxx <tracecompass-dev-bounces@xxxxxxxxxxx> on behalf of David Wootton <dwootton@xxxxxxxxxx>
Sent: Monday, January 7,
2019 10:42:48 AM
To: tracecompass developer
discussions
Subject: Re:
[tracecompass-dev] Fw: Heap storage consumption for traces
Matthew.
That's good news. Will this be in the Simrel 2019-03
release? Will there be a Trace Compass repository site I can
point to at some point in time to give this a quick look?
Thanks to Genevieve for the optimization work.
Dave
Matthew Khouzam ---01/07/2019
10:20:52 AM---Hey, Dave, Happy Holidays! Hope Santa was good
for you, she was for me, her name is Genevieve and sh
From: Matthew Khouzam
<matthew.khouzam@xxxxxxxxxxxx>
To: tracecompass
developer discussions <tracecompass-dev@xxxxxxxxxxx>
Date: 01/07/2019 10:20
AM
Subject: Re:
[tracecompass-dev] Fw: Heap storage consumption for traces
Sent by: tracecompass-dev-bounces@xxxxxxxxxxx
Hey, Dave,
Happy Holidays! Hope Santa was good for you, she was for me,
her name is Genevieve and she really optimized the callgraph
over the vacations. I think you will be pleasantly surprised
with the results. 😊
BR
Matthew
From: tracecompass-dev-bounces@xxxxxxxxxxx <tracecompass-dev-bounces@xxxxxxxxxxx> on behalf of David Wootton <dwootton@xxxxxxxxxx>
Sent: Thursday, January 3,
2019 7:20:16 AM
To: tracecompass developer
discussions
Subject:
[tracecompass-dev] Fw: Heap storage consumption for traces
Matthew
I hope you had a good holiday. I'm following up on this now
that I am back at work. Do you need anything from me at this
point, and how can I get it to you?
Dave
----- Forwarded by David Wootton/Poughkeepsie/Contr/IBM on
01/03/2019 07:18 AM -----
From: David
Wootton/Poughkeepsie/Contr/IBM
To: tracecompass
developer discussions <tracecompass-dev@xxxxxxxxxxx>
Date: 12/05/2018 02:28
PM
Subject: Re:
[tracecompass-dev] Heap storage consumption for traces
Matthew
I think I can send you my heap dump, but I need to check.
It's about 200MB and I don't have any public site I can
upload it to. How do I get it to you?
I'm not really familiar with VisualVM so I'm not sure how I
check what owns an object.
I am a little surprised that I seem to only have 52
TmfStateValue objects when I have 4000 ITmfEventField
objects. Maybe my placement for the printf call is wrong and
most of the objects have been released, I placed it in my
class overriding the call stack view, in the traceOpened
method. But I am using a hash table to keep only a single
instance of a value.
If I understand, in TraceCompass 4.1, at the point where I
create a TmfStateValue to wrap an object (String, Long,
Integer, etc), I can now just use that object itself in the
call to the state builder pushAttribute call. Is that
correct?
Happy holidays
Dave
Matthew Khouzam ---12/05/2018
10:30:53 AM---Hi Dave, these are some pretty cool findings.
Now one thing to keep in mind, we have several caches,
From: Matthew Khouzam
<matthew.khouzam@xxxxxxxxxxxx>
To: tracecompass
developer discussions <tracecompass-dev@xxxxxxxxxxx>
Date: 12/05/2018 10:30
AM
Subject: Re:
[tracecompass-dev] Heap storage consumption for traces
Sent by: tracecompass-dev-bounces@xxxxxxxxxxx
Hi Dave,
these are some pretty cool findings. Now one thing to keep
in mind, we have several caches, so they do trigger visualVM
and rightly so. Can you look up who is the owner of the
mapEntries? Maybe we have a leak with a manager or
something.
Also, please note, tmfstatevalues are on the road to
deprecation. The state system now handles objects, so you
can save a few megabytes of useless pointers by just using
these objects. 😊
If you are allowed to share your heapdump, we can look at
that and try to optimize a bit here.
There is finally a possibility, but not a large one. Does
your EventField have a backreference to the event, which
contains a map of its fields? That would be a prime
candidate for where things can go wrong.
BR,
Matthew. (Is it too soon to wish a happy holiday season?)
From: tracecompass-dev-bounces@xxxxxxxxxxx <tracecompass-dev-bounces@xxxxxxxxxxx> on behalf of David Wootton <dwootton@xxxxxxxxxx>
Sent: Wednesday, December
5, 2018 8:58:57 AM
To: tracecompass developer
discussions
Subject:
[tracecompass-dev] Heap storage consumption for traces
A while back, Matthew and I discussed heap storage
utilization by traces. My traces have a number of attributes
associated with event, with the number of attributes varying
by type of trace. I create ITmfEventField and TmfStateValue
objects for each attribute.
I have a trace with about 88,000 events. In that trace, I
ended up with close to 800,000 ITmfEventField and 500,000
TmfStateValue objects.
I created a hash table for ITmfEventField objects that
contained a single instance for each unique field name and
value pair. In this trace, I ended up with about 4,000
unique ITmfEventField objects.
I created a hash table for the TmfStateValue objects and end
up with 52 unique objects.
I ran my code with VisialVM, and I noticed that the top
class for heap usage was
org.eclipse.tracecompass.internal.statesystem.core.backend.historytree.HTInterval.
From one run to the next the total number of objects varies
somewhere between 1.5 million and 2.5 million. If I click
the Perform-GC button in VisualVM, the count drops to around
700,000, fairly consistently.
I also see a large number of instances of
com.google.com.collect.ImmutableMapEntry (355,000),
com.google.com.collect.ImmutableMapEntry[] (178,000) classes
and
com.google.com.collect.ImmutableMapEntry$NonTerminalImmutableMapEntry
(355,000)
Looking at the reference pane in VisualVM for a few (<
10) of each of these classes, it looks like the
ImmutableMapEntry and
ImmutableMapEntry$nonTerminalImmutableMap objects might be
related to StateItem objects and the ImmutableMapEntry[]
objects might be related to ITmfEventField objects.
I'm wondering if it is possible to use a hash table to
maintain single copies of unique instances of those objects
or whether the data in them guarantees each instance is
unique. If it is possible, then is the performance tradeoff
for ushing a hash table worth it?
Dave_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password,
or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password,
or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password,
or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
|