Re: [mat-dev] Memory Analyzer 1.12 and Eclipse 2021-06

[Jason Koch <jkoch@xxxxxxxxxxx>]

You make a good point on GarbageCleaner. In practice I have not come
across this issue, but am now seeing GarbageCleaner as the limit as I
experiment off-heap index generation. I introduced the
four-layer-inline as earlier iterations would end up in some cases
with a stack overflow and this ended up as a good heuristic to avoid
overflows it (and has been fine for our apps). Perhaps a simple
sort/uniq/heap as the recursion progresses would be enough to resolve
it. I will have a think about this; there is probably a cleaner
iterative solution that can address both concerns.

On Sat, May 1, 2021 at 9:49 AM Andrew Johnson <andrew_johnson@xxxxxxxxxx> wrote:
>
> > OK, I did the investigation on the retained set issue, and I was a
> > little hasty; it was actually some code I had in development on my end
> > related to memory mapping changes. My 'blocker' is safe to ignore.
> >
> > On Fri, Apr 30, 2021 at 10:01 AM Jason Koch <jkoch@xxxxxxxxxxx> wrote:
> > >
> > > Agree w/ plan for Java 11.
> > > - From my pov the GarbageCleaner improvements are isolated enough that
> > > if we want to, we can ship them.
> > >
> > > - The memory mapping is technically viable but I think it would be
> > > good to get some other opinions, think about a strategy for our users,
> > > etc. This would require a shift for user configuration between heap
> > > and disk IO. It might need some heuristics to automatically decide
> > > which to use, or feature flags to allow user control.
> > >
> > > - I see one issue that (imo) is a blocker, I think this needs some
> > > further investigation, as I am seeing heaps with incomplete retained
> > > sets especially from threads in
> > > flight:https://bugs.eclipse.org/bugs/show_bug.cgi?id=569223
> > >
> > > Thanks
> > > Jason
> Thanks for the update on Bug 569223.
> I'm still looking at the garbage cleaner improvements but I don't think
> they are quite ready yet.
> The object marker uses the Java Fork/Join library, which is good, but I
> don't know the worst case memory consumption. The previous marker had a
> natural limit of one int array entry for each heap object. The new one
> stores an outbound array as a work unit, which has the advantage of
> packing more work into one ForkJoin item. However if there was a heap dump
> with lots of object arrays 4 levels deep from the GC roots then it would
> recurse to that level, then queue all those outbounds arrays, which
> potentially could be much bigger than just the number of objects in the
> snapshot.
>
> I got a bit sidetracked by Bug 573258
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=573258 as I kept on finding
> other nice to have items.
>
> I think this is a worthwhile enhancement to the component report.
> There are three additional sections for:
>  Zero-Length Arrays
>  Array Fill Ratios
>  Primitive Arrays with a Constant Value
> This involved some extra improvements in adding highlighting to findings
> of wasted memory, and warning triangle tags which go up to the component
> level to make it clear which sections to focus on. Now that is working we
> can add more sections if anyone has some good ideas.
> The primitive arrays with constant value is now a quantized report
> grouping objects of the same length with same repeated value.
> The fill ratio queries now directly show the wasted bytes in the backing
> arrays.
> The array size query now shows the array length as well as the array size
> The collection/array queries now show updates for which type of object
> they are operating on.
> I have suppressed messages about unrecognized collections after 20
> messages per type.
> The lazy loading of array lengths (like field are lazy loaded) makes it
> quick to do a getObject() and view its type, retained size etc. which is
> useful for OQL and calculating the members of a component.
> The quantize queries now also operate on MAT Bytes as well as long and
> double.
> A possible problem with adding the 'Zero-Length Arrays' and 'Array Fill
> Ratios' is that every array object ID will end up in a list inside the
> query result. Sometimes it is used - for the links 'First 10 of 46
> objects' but most are not, but they use memory until the HTML has been
> built.
>
> Do try out the updated Top Components Report.
>
> Krum, it it Eclipse 2021-06 M2 this week, are you able to submit what we
> have so far, say on Tuesday May 4 2021, ready for M2+3?
>
> --
> Andrew Johnson
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
> _______________________________________________
> mat-dev mailing list
> mat-dev@xxxxxxxxxxx
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/mat-dev