Re: [glassfish-dev] Tracking usage data for EE4J working group CI cloud

[Scott Marlow <smarlow@xxxxxxxxxx>]

I added some notes to 
https://github.com/eclipse-ee4j/glassfish/issues/23191#issuecomment-702454944 
with notes from debugging.  I am not sure of why the 
ClassNotFoundException occurs and that should be fixed, if someone can 
figure out why the classpath might be wrong sometimes or how to diagnose 
that problem.

More importantly, IMO, GlassFish 
com.sun.enterprise.v3.server.AppServerStartup.run() should return an 
indication if the server process should terminate when serious problems 
occur so that com.sun.enterprise.v3.server.AppServerStartup.doStart() 
can know whether to start the nondaemon thread or terminate immediately. 
 I assume that is hard with multiple GlassFish run levels and I don't 
know the GlassFish internals, so someone else should speak up here.

To me, any startup failure that is important should cause the server to 
terminate.

Is "java.lang.ClassNotFoundException: 
org.glassfish.flashlight.MonitoringRuntimeDataRegistry not found by 
org.glassfish.main.admin.monitoring-core [190]" a serious failure?

If yes, it would be nice to have an error logged instead of having to 
enable --verbose to see it as well as have a way for the server to 
terminate instead of just being incompletely started (IMO).

By the way, the errors seem to be logged to the console and to 
glassfish6/glassfish/domains/domain1/logs/server.log. :-)

Scott

On 10/1/20 6:42 PM, Scott Marlow wrote:
> On 10/1/20 6:32 PM, arjan tijms wrote:
> > Hi,
> >
> > Just some other comment, when GF fails to start up during those tests 
> > for 
> > https://ci.eclipse.org/glassfish/view/GlassFish/job/glassfish_build-and-test-using-jenkinsfile 
> > there's no exception whatsoever in the logs. Those exceptions 
> > you report may possibly be of a different issue entirely.
>
> I only see the exceptions if --verbose option is used to start GlassFish.
>
> With or without --verbose, the last thing that I see is "Server shutdown 
> initiated" in the server.log (even though I didn't initiate server 
> shutdown.
>
> Kind regards,
> Scott
>
> > Kind regards,
> > Arjan
> >
> > On Thu, Oct 1, 2020 at 11:41 PM Scott Marlow <smarlow@xxxxxxxxxx 
> > <mailto:smarlow@xxxxxxxxxx>> wrote:
> >
> >
> >
> >     On 10/1/20 3:50 PM, Scott Marlow wrote:
> >      >
> >      >
> >      > On 10/1/20 12:34 PM, Scott Marlow wrote:
> >      >>
> >      >>
> >      >> On 10/1/20 12:04 PM, arjan tijms wrote:
> >      >>> Hi,
> >      >>>
> >      >>> A few remarks here: just repeating the startup of GF doesn't
> >     seem to
> >      >>> work. The test jobs for the GF build themselves already do
> >     that. It
> >      >>> repeats the entire test 3 times. I've never ever seen it work
> >     after
> >      >>> the second or third repeat.
> >      >>
> >      >> Are you seeing the same
> >      >> https://github.com/eclipse-ee4j/glassfish/issues/23191 (GF
> >     server.log
> >      >> contains `Server shutdown initiated`)?
> >      >
> >      > Note that the issue actual does contain the server --verbose
> >     output via
> >      >
> >     
> > https://github.com/eclipse-ee4j/glassfish/files/5130976/verboselogs.txt
> >      >
> >      > 1.  What does the java.lang.RuntimeException:
> >      > java.util.concurrent.ExecutionException:
> >     java.lang.NoClassDefFoundError:
> >      > Lorg/glassfish/flashlight/MonitoringRuntimeDataRegistry failure 
> > mean?
> >      >
> >      > 2.  Is the MonitoringRuntimeDataRegistry.class missing from a
> >     classpath?
> >      >
> >      > 3.  I tried -Dorg.glassfish.startupThreadPolicy=USE_NO_THREADS in
> >      > glassfish/domains/domain1/config/domain.xml but that didn't seem
> >     to make
> >      > a difference (nor did `rm -rf 
> > glassfish/domains/domain1/osgi-cache`).
> >      >
> >      > 4.  In the email thread, one of the last comments was from 
> > Steve via
> >      > https://www.eclipse.org/lists/glassfish-dev/msg00514.html:
> >      > "
> >      > It's strange that it thinks it needs to update the bundles.
> >      > "
> >      >
> >      > I did confirm that GlassFish5 still works fine on my local 
> > machine.
> >      >
> >      > Any other suggestions of what to try?
> >
> >     5. internal-api.jar contains the
> >     org.glassfish.flashlight.MonitoringRuntimeDataRegistry class that we
> >     get
> >     the NoClassDefFoundError failure for.
> >
> >     monitoring-core.jar depends on the
> >     org.glassfish.flashlight.MonitoringRuntimeDataRegistry class which if
> >     accessed through the monitoring-core.jar would likely be okay, since
> >     monitoring-core.jar contains MANFEST.MF with internal-api.jar 
> > included
> >     in the Class-Path.
> >
> >     However, given that we see a java.util.concurrent.ExecutionException:
> >     java.lang.NoClassDefFoundError:
> >     Lorg/glassfish/flashlight/MonitoringRuntimeDataRegistry that is
> >     followed
> >     by GlassFish 6 terminating, does that mean the wrong classpath is 
> > used
> >     sometimes due to a race condition?
> >
> >     https://gist.github.com/scottmarlow/0e0ce78a9da4519b416a63490eecdd06
> >     contains the MANIFEST.MF contents for both internal-api.jar +
> >     monitoring-core.jar.
> >
> >     Scott
> >
> >     _______________________________________________
> >     glassfish-dev mailing list
> >     glassfish-dev@xxxxxxxxxxx <mailto:glassfish-dev@xxxxxxxxxxx>
> >     To unsubscribe from this list, visit
> >     https://www.eclipse.org/mailman/listinfo/glassfish-dev
> >
> >
> > _______________________________________________
> > glassfish-dev mailing list
> > glassfish-dev@xxxxxxxxxxx
> > To unsubscribe from this list, visit 
> > https://www.eclipse.org/mailman/listinfo/glassfish-dev