| Re: [milo-dev] Netty LEAK message |
Additional information.
The operating environment is as follows.
OS: CentOS Linux release 7.3.1611 (Core)
CPUs: 4
Mem: 4 GB
Java: OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)
This server from 6 clients
- Publishing interval 200 (msec)
- Sampling interval 100 (msec)
- NodeId items 350
- Number of concurrent clients 6
Requests are accepted with the above settings.
As changed points from last night, concurrent GC is enabled and it is running.
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+UseParNewGC
As a result, the increasing tendency of Used Tenured Heap has drastically decreased.
As a feature of concurrent GC, usage of Old area may be easy to raise, but in this case
it decreased conversely.
In addition, the Used Young Heap for Young Generation has remained at around 10%,
and has remained constant at the moment.
Even if OutOfMemoryError occurs, it will be about a week ahead if it keeps going
like this.
<ishida_shigeru@xxxxxxxxxxx> wrote, Tue, 19 Dec 2017 13:39:39 +0900 (JST)
> OK.
>
> From yesterday I updated Milo to 0.1.6 and Netty to 4.0.54 and started.
> After about 9 hours,
>
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> The log above has come up, and it look like LEAK at the GC log in GCView.
> Anyways I changed the parameters of memory & GC and restarted it.
>
> I will continue to investigate.
>
> <kevinherron@xxxxxxxxx> wrote, Mon, 18 Dec 2017 06:31:39 -0800
>
> > Yes, you're right, that buffer would leak if the exception was thrown.
> >
> > I've already released 0.1.6 so I'll fix it in 0.2.0. Fortunately I don't
> > think that exception is likely to ever be thrown, so it would only leak
> > under the rarest of circumstances.
> >
> > On Mon, Dec 18, 2017 at 12:14 AM, Shigeru Ishida <ishida_shigeru@xxxxxxxxxxx
> > > wrote:
> >
> > > Hi Kevin,
> > >
> > > I saw the code of Milo version 0.1.6. There is one point to worry about.
> > >
> > > ChunkDecoder#decryptChunk()
> > > L177: ByteBuf plainTextBuffer = BufferUtil.buffer(plainTextBufferSize);
> > >
> > > The plainTextBuffer obtained above does not mean that release() will not
> > > be called when an exception is thrown in L200?
> > >
> > > I am sorry if I misunderstood.
> > >
> > > Regards,
> > >
> > > --Shigeru
> > >
> > > <kevinherron@xxxxxxxxx> wrote, Sat, 16 Dec 2017 10:51:16 -0800
> > >
> > > > Leak fixes have been merged into master (already fixed in dev/0.2.x).
> > > >
> > > > I'm releasing 0.1.6 this weekend. It will be the last of the 0.1 series.
> > > >
> > > > 0.2.0 will be released by the end of next week, if not this weekend.
> > > >
> > > > On Thu, Dec 14, 2017 at 7:19 AM, Kevin Herron <kevinherron@xxxxxxxxx>
> > > wrote:
> > > >
> > > > > > As one goal, I expect to run it continuously and stably for about
> > > two
> > > > > months.
> > > > >
> > > > > Yes, of course. I would expect it to run stable indefinitely. Thanks
> > > for
> > > > > testing.
> > > > >
> > > > > The focus of the 0.3 release is going to be on the server SDK.
> > > > >
> > > > > On Wed, Dec 13, 2017 at 8:43 PM, Shigeru Ishida <
> > > > > ishida_shigeru@xxxxxxxxxxx> wrote:
> > > > >
> > > > >> One day passed, it does not look like LEAK from the GC log.
> > > > >> I changed it to output the GC log in detail and restarted it.
> > > > >>
> > > > >> As one goal, I expect to run it continuously and stably for about two
> > > > >> months.
> > > > >>
> > > > >> <kevinherron@xxxxxxxxx> wrote, Wed, 13 Dec 2017 05:21:01 -0800
> > > > >>
> > > > >> > Okay, interesting.
> > > > >> >
> > > > >> > The other place leaks were fixed was the re-implementation of
> > > > >> ChunkEncoder
> > > > >> > and ChunkDecoder in dev/0.2.x.
> > > > >> >
> > > > >> > You can see when encoding/decoding fails they clean up:
> > > > >> >
> > > > >> > https://github.com/eclipse/milo/blob/ 37bee220ae53026db87328c
> > > > >> fbb0b664671bff071/opc-ua-stack/stack-core/src/main/
> > > > >> java/org/eclipse/milo/opcua/stack/core/channel/
> > > ChunkEncoder.java#L86-L93
> > > > >> > https://github.com/eclipse/milo/blob/ 37bee220ae53026db87328c
> > > > >> fbb0b664671bff071/opc-ua-stack/stack-core/src/main/
> > > > >> java/org/eclipse/milo/opcua/stack/core/channel/
> > > ChunkDecoder.java#L82-L92
> > > > >> >
> > > > >> > This won't be a straight-forward cherry pick like the other changes.
> > > > >> >
> > > > >> > Keep me updated.
> > > > >> >
> > > > >> >
> > > > >> > On Tue, Dec 12, 2017 at 10:03 PM, Shigeru Ishida <
> > > > >> ishida_shigeru@xxxxxxxxxxx
> > > > >> > > wrote:
> > > > >> >
> > > > >> > > Hi Kevin,
> > > > >> > >
> > > > >> > > Merge the following commits in dev/0.2.x,
> > > > >> > >
> > > > >> > > - 2017/12/09 c1b6978
> > > > >> > > - 2017/12/13 e4e5915
> > > > >> > >
> > > > >> > > Furthermore, I replaced ChunkDecoder.java from
> > > > >> > > https://github.com/eclipse/milo/tree/release-buffers and
> > > executed it.
> > > > >> > >
> > > > >> > > The JVM options are the same. I also set the GC log.
> > > > >> > >
> > > > >> > > -Dio.netty.leakDetection.maxRecords=1000
> > > > >> > > -Dio.netty.leakDetection.level=paranoid
> > > > >> > >
> > > > >> > > Also, the setting of the concurrent clients is the same.
> > > > >> > >
> > > > >> > > - Publishing interval 200 (msec)
> > > > >> > > - Sampling interval 100 (msec)
> > > > >> > > - NodeId items 350
> > > > >> > > - Number of concurrent clients 6
> > > > >> > >
> > > > >> > > When 1.5 hours passed, the increasing trend of GC's heap area
> > > remained
> > > > >> > > as usual, and it seems that LEAK message seems to be coming out in
> > > > >> about
> > > > >> > > 4 hours from the past experience if this trend continues.
> > > > >> > >
> > > > >> > > Therefore, when changing from Netty 4.1.4 to 4.0.53 (the latest
> > > > >> version of
> > > > >> > > 4.0.x),
> > > > >> > > the occurrence frequency of Scavenge GC / Full GC decreased
> > > > >> drastically,
> > > > >> > > and the heap area increased by about 8 minutes It was stopped.
> > > > >> > >
> > > > >> > > I have not seen it for about 1.5 hours yet, but it feels good.
> > > > >> > >
> > > > >> > > It is early to state the conclusion, so let's go on continuously
> > > > >> > > as it is and see the situation.
> > > > >> > >
> > > > >> > > Furthermore, since the mail became too big, I cut down quoted
> > > mails
> > > > >> > > other than the nearest.
> > > > >> > >
> > > > >> > > Regards,
> > > > >> > >
> > > > >> > > --Shigeru
> > > > >> > >
> > > > >> > > <kevinherron@xxxxxxxxx> wrote, Tue, 12 Dec 2017 17:16:37 -0800
> > > > >> > >
> > > > >> > > > I think I've squashed all of the leaks that occur under
> > > exceptional
> > > > >> > > > circumstances in the dev/0.2.x branch:
> > > > >> > > > https://github.com/eclipse/milo/tree/dev/0.2.x
> > > > >> > > >
> > > > >> > > > It's a little difficult to cherry pick the fixes because the
> > > 0.2.x
> > > > >> branch
> > > > >> > > > has significant changes.
> > > > >> > > _______________________________________________
> > > > >> > > milo-dev mailing list
> > > > >> > > milo-dev@xxxxxxxxxxx
> > > > >> > > To change your delivery options, retrieve your password, or
> > > > >> unsubscribe
> > > > >> > > from this list, visit
> > > > >> > > https://dev.eclipse.org/mailman/listinfo/milo-dev
> > > > >> _______________________________________________
> > > > >> milo-dev mailing list
> > > > >> milo-dev@xxxxxxxxxxx
> > > > >> To change your delivery options, retrieve your password, or
> > > unsubscribe
> > > > >> from this list, visit
> > > > >> https://dev.eclipse.org/mailman/listinfo/milo-dev
> > > _______________________________________________
> > > milo-dev mailing list
> > > milo-dev@xxxxxxxxxxx
> > > To change your delivery options, retrieve your password, or unsubscribe
> > > from this list, visit
> > > https://dev.eclipse.org/mailman/listinfo/milo-dev
_______________________________________________
milo-dev mailing list
milo-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/milo-dev