Re: [hono-dev] Using G1 garbage collector

["Hudalla Kai (INST/ECS4)" <kai.hudalla@xxxxxxxxxxxx>]

Sorry for the encrypted mail, I am still learning to tame Thunderbird ;-)

What I wanted to say is the following:

We had an interesting discussion on the Hono IRC channel (#eclipse-hono)
this morning regarding this issue and I would like to "preserve" a
transcript of it by sharing on this list ...

09:01:12 - ppatiern: sophokles: Kai let me to introduce Francesco
(franz1981) our performance expert !
09:01:35 - franz1981: <ppatiern> thanks Paolo!
09:01:40 - ppatiern: I forwarded him your message from the hono dev list
09:01:48 - sophokles: franz1981: Hi Francesco, nice to meet you :-)
09:02:03 - sophokles: thanks for taking the time to help us out!
09:02:13 - franz1981: <sophokles> ywc!!!
09:02:49 - sophokles: franz1981: any initial thoughts on the issue?
09:03:17 - ppatiern: Franz was asking if you raised the question due to
some performance problems you had
09:03:29 - sophokles: or would you rather take your time and send a
reply on the list (would be even better FMPOV)?
09:04:05 - franz1981: <sophokles> Exectly what Paolo said...and if I
feel to need some time to think about it I will for sure :)
09:04:32 - sophokles: well, due to the nature of our use case
(messaging), we are constantly creating a lot of objects with a very
short lifespan
09:04:58 - sophokles: so my understanding is that these objects will
pile up  quickly before getting GCed using the standard mark and sweep GC
09:06:30 - sophokles: Using the MaS GC we will probably have the classic
chainsaw memory usage graph with potentially long GC runs once in a while
09:06:53 - sophokles: My hope is that we get a better (more constant and
predictable) GC behavior with the G1 GC
09:07:21 - sophokles: does that make sense?
09:08:45 - franz1981: <sophokles> In theory yes, but it mostly depend on
your expectation about: 1) when the collection of such short lived
objects will happen 2) what does it mean being short lived from the GC
point of view
09:10:36 - sophokles: ad 1) what I would like to prevent is having "full
stops" for  one or more seconds every 10  - 30 secs during which no
messages can be processed
09:11:44 - sophokles: ad 2) the objects created (representing the
messages being forwarded) will have a life span of several micro seconds
(if at all)
09:13:31 - franz1981: <sophokles> ok so: re 1) it could happen with G1
too (dependently on different factors, like the numbers of created
objects vs heap size)
09:14:56 - franz1981: <sophokles> and re 2) the GC measure lifecycle of
object using collection as aging mechanism and a collection will happen
always when the space (where the instance is living) need to be
reclaimed because is full (or near to a specified limit)
09:15:24 - franz1981: <sophokles> So to make it more actionable...
09:16:30 - franz1981: <sophokles> if you have a small eden you risk that
due to the frequent minor Collection the object will became "old",
forcing G1 to perform Full GCs
09:17:28 - franz1981: <sophokles> if you have a too big eden the
effectiveness (pause time) is dependent on the number of instances
created and how much they are scattered in the heap
09:17:55 - franz1981: <sophokles> it is just a matter to try different
loads and tune the eden size
09:18:14 - franz1981: <sophokles> then you can see if the pauses are ok
with your SLA
09:18:48 - franz1981: <sophokles> but just be aware that the throughput
of the application a lil' less than with the other GCs
09:18:52 - franz1981: <sophokles> makes sense?
09:20:29 - sophokles: I have to admit that I do not know the meaning of
the "eden size". However, I can dig into that offline. My more general
question is if using the G1 GC instead of the standard MaS has the
potential of resulting in a better (more predictable) GC behavior for
our use case.
09:21:03 - sophokles: Finding the "best" param values (e.g. for eden
size) is then, as you said, probably a matter of experimentation/load
testing
09:22:26 - franz1981: <sophokles> the short answer is yes, if the tuning
is ok, while if not, it could be actually worse, because when G1 is
overloaded it perform Full GC pauses with only 1 thread
09:23:48 - sophokles: Ok, so the trick is to find param values that
prevent the G1 GC from doing Full GC runs in the first place, right?
09:24:12 - franz1981: <sophokles> exactly, that's the important part...
09:24:50 - sophokles: any hints/links to resources that can help us
finding/determining/tuning the relevant params and their values?
09:25:17 - franz1981: <sophokles>Yes, it is a long but very valuable
one:
http://product.hubspot.com/blog/g1gc-fundamentals-lessons-from-taming-garbage-collection
09:25:55 - sophokles: franz1981: fantastic, thanks a lot for your time,
this was really helpful :-)
09:26:48 - franz1981: <sophokles> ywc!! And just a last advice: the G1
tends to be less effective with about >15/20 GB heap
09:27:10 - sysexcontrol [~kf@xxxxxxxxxxxxxxxxxxx] entered the room.
09:27:41 - franz1981: <sophokles> If for any reason this would became
the size of the heap you need, have a look to this:
https://wiki.openjdk.java.net/display/shenandoah/Main
09:28:06 - sophokles: franz1981: I think we will mostly deal with much
smaller memory/heap sizes. In the end, we are talking about "micro
services" ;-)
09:28:09 - franz1981: <sophokles> it will be available soon, and it has
been already tested with Infinispan with success
09:28:38 - franz1981: <sophokles> makes sense!!

HTH


On 31.07.2017 09:59, Hudalla Kai (INST/ECS4) wrote:
> Hi,
> 
> in recent discussions with people conducting some basic load tests the issue came up whether it would be a good idea to use the G1 garbage collector by default for the Hono components.
> 
> FMPOV this seems to make a lot of sense since it seems to be better suited for our use case of constantly creating huge numbers of obejct instances having a very short life span, i.e. which can be garbage collected as soon as the messages they represent have been forwarded downstream to the next component.
> 
> Any thoughts/opinions?
> 
> Mit freundlichen Grüßen / Best regards
> 
> Kai Hudalla
> Chief Software Architect
> 
> Bosch Software Innovations GmbH
> Schöneberger Ufer 89-91
> 10785 Berlin
> GERMANY
> www.bosch-si.com
> 
> Registered office: Berlin, Register court: Amtsgericht Charlottenburg, HRB 148411 B;
> Executives: Dr.-Ing. Rainer Kallenbach, Michael Hahn
> _______________________________________________
> hono-dev mailing list
> hono-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/hono-dev
> 

-- 
Mit freundlichen Grüßen / Best regards

Kai Hudalla
Chief Software Architect

Bosch Software Innovations GmbH
Schöneberger Ufer 89-91
10785 Berlin
GERMANY
www.bosch-si.com

Registered Office: Berlin, Registration Court: Amtsgericht
Charlottenburg; HRB 148411 B
Chairman of the Supervisory Board: Dr.-Ing. Thorsten Lücke; Managing
Directors: Dr.-Ing. Rainer Kallenbach, Michael Hahn