Re: [glassfish-dev] Tracking usage data for EE4J working group CI cloud

[Scott Marlow <smarlow@xxxxxxxxxx>]

I started 
https://ci.eclipse.org/jakartaee-tck/job/jakartaee-tck-scottmarlow/job/glassfish_start_twice/2/ 
running with the workaround suggested on 
https://github.com/eclipse-ee4j/glassfish/issues/23191#issuecomment-708977368

The workaround changes are added to docker/run_jakartaeetck.sh in 
https://github.com/scottmarlow/jakartaee-tck/tree/glassfish_start_twice.

Basically, the workaround moves the monitoring-core.jar out of GlassFish 
and starts up GlassFish which will fail but it somehow works around the 
startup failure that we occasionally see on Jenkins and always see on my 
local computer.  After stopping GlassFish, we will then move the 
monitoring-core.jar file back into place for the next startup of GlassFish.

If this test works, we can decide if we want to move the same change 
into the Standalone TCK scripts that startup GlassFish as well.

Scott

On 10/1/20 12:04 PM, arjan tijms wrote:
> Hi,
>
> A few remarks here: just repeating the startup of GF doesn't seem to 
> work. The test jobs for the GF build themselves already do that. It 
> repeats the entire test 3 times. I've never ever seen it work after the 
> second or third repeat.
>
> Yet, starting the entire process does get it to pass.
>
> This leads me to believe it's something in the Kubernetes pod and/or 
> Docker container that gets corrupted. Once this corruption happens, GF 
> will always fail to startup.
>
> Please see this issue about this: 
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=561229 
> <https://bugs.eclipse.org/bugs/show_bug.cgi?id=561229>
>
> It discusses the problem from the Jenkins/Kubernetes point of view. 
> There's a few odd things in the Kubernetes setup which may have to be 
> looked at first.
>
> Kind regards,
> Arjan Tijms
>
>
>
>
>
>
>
>
>
>
> On Thu, Oct 1, 2020 at 5:41 PM Scott Marlow <smarlow@xxxxxxxxxx 
> <mailto:smarlow@xxxxxxxxxx>> wrote:
>
>
>
>     On 10/1/20 10:48 AM, Ed Bratt wrote:
>      > I wasn't intending to point any fingers at the stability
>     observations
>      > you made -- only to observe that we want to focus on improving the
>      > reliability so that we don't need to rely on re-runs, or waits,
>     or other
>      > symptomatic only type fixes.
>
>     I agree 100%!  I don't think that we need to make the memory tuning
>     changes just yet, as they will not improve reliability.
>
>      > One day, I'd like to see that we can
>      > initiate several test runs simultaneously -- and they all reliably
>      > complete -- perhaps taking more clock-time, but reliably and
>      > consistently repeating.
>
>     +1
>
>      >
>      > Ideally, we could fill this compute pipeline with running and
>     waiting
>      > tasks and be confident that the system will reliably produce,
>     consistent
>      > results. It sounds like, for some unknown cause, we aren't there
>     yet.
>      > Getting to the root cause of this would be my priority (and we don't
>      > know if that's a GlassFish issue, an infrastructure issue or even
>      > something else). But, I'm not actually doing the work so, it's
>     just my
>      > opinion.
>
>     +1
>
>     Part of the original discussion related to
>     https://github.com/eclipse-ee4j/glassfish/issues/23191
>     <https://github.com/eclipse-ee4j/glassfish/issues/23191> suggested
>     adding
>     a --verbose --debug options which I tried on my local machine (as I can
>     reproduce a similar symptom that could be the same as on CI), I would
>     see `org.glassfish.flashlight.MonitoringRuntimeDataRegistry not
>     found by
>     org.glassfish.main.admin.monitoring-core` as shown on
>     https://gist.github.com/scottmarlow/eec11ca74b99d021346b270fa29ce4fa
>     <https://gist.github.com/scottmarlow/eec11ca74b99d021346b270fa29ce4fa>
>     (which has the full server output including error exceptions).
>
>     I'm thinking that we could try to reproduce the failure on Jenkins with
>     the --verbose --debug options so that we see the actual cause on
>     Jenkins.  Perhaps it will be the same exception call stack as I see
>     locally.
>
>     I don't know the code involved in the
>     https://gist.github.com/scottmarlow/eec11ca74b99d021346b270fa29ce4fa
>     <https://gist.github.com/scottmarlow/eec11ca74b99d021346b270fa29ce4fa>
>     exception call stack enough to debug and find the cause.  If you open
>     this gist, please search for `java.lang.ClassNotFoundException:
>     org.glassfish.flashlight.MonitoringRuntimeDataRegistry not found by
>     org.glassfish.main.admin.monitoring-core` which kind of seemed like a
>     startup race condition but I have no idea really.
>
>     Scott
>
>     _______________________________________________
>     glassfish-dev mailing list
>     glassfish-dev@xxxxxxxxxxx <mailto:glassfish-dev@xxxxxxxxxxx>
>     To unsubscribe from this list, visit
>     https://www.eclipse.org/mailman/listinfo/glassfish-dev
>     <https://www.eclipse.org/mailman/listinfo/glassfish-dev>
>
>
> _______________________________________________
> glassfish-dev mailing list
> glassfish-dev@xxxxxxxxxxx
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/glassfish-dev