I’ve raised an issue on the GlassFish project to get our Jenkins master build job to upload an artefact to the Eclipse download site. See
https://github.com/eclipse-ee4j/glassfish/issues/23022 I’ll update this list when that issue is done and we have a download there.
AFAIK GlassFish uses these jobs to run the CTS
https://ci.eclipse.org/glassfish/view/cts/ But Ed/Dmitry and their team set those jobs up so will be more familiar.
Steve
From: jakartaee-tck-dev-bounces@xxxxxxxxxxx <jakartaee-tck-dev-bounces@xxxxxxxxxxx>
On Behalf Of Scott Marlow
Sent: 20 May 2020 02:47
To: Ed Bratt <ed.bratt@xxxxxxxxxx>
Cc: jakartaee-tck developer discussions <jakartaee-tck-dev@xxxxxxxxxxx>
Subject: Re: [jakartaee-tck-dev] Weekly building of Jakarta Platform Stand-alone TCKs...
Thanks Ed, good to raise validation (and other things) for discussion here! :) More inline below...
Some additional comments and ideas:
I think that some of the TCK build paths are failing -- or, what I'm actually observing is that some TCKs are not at the same version as their specification (i.e. annotations-tck is either not updated, or the build job is still creating files with the name
'1.3.0' in their string. I presume the Annotations TCK will become 2.x.x when it's updated. Some of the TCK zip files are updated. WebSocket, for example is 2.0.0 so I believe that is actually getting some attention). We may find that some of these build paths
are failing along the way and simply not generating a zip.
A useful metric could be simply that these zip files are being produced and have the correct version IDs. (By a count I did last week, there were 6 zip files that seemed to be updated using this metric. There could be more this week, I haven't checked yet.)
In addition when you look at the Jenkins job under JUnit Reports (e.g.
here), you will see a few TCK entry lines (12 right now) -- nearly all are failing.
These test jobs use GlassFish builds and will perform a validation on each stand-alone TCK. I hesitate to call that compatibility validation, because there was concern in EE 8 about how those tests are run -- but it is a potential comparative point.
Does the GlassFish project also run each stand-alone TCK, as well? Or rely on the https://ci.eclipse.org/jakartaee-tck/job/standalonetck-nightly-build-run-master for
validation? If we do rely on standalonetck-nightly-build-run-master, one low cost improvement, is to have TCK failures trigger email notification to be sent to those interested (or we could have a new public mailing list that TCK failures are sent to, if
others would join such a potentially noisy mailing list :). We currently do this for the nightly build of the Platform TCK.
Regardless,
there were 23 entries in EE 8, just under 19K tests that passed. These numbers will change for EE 9 due to pruning. But we can track this evolution as the work progresses.
Should we wait until we are running with a daily GlassFish master (6.0) build to communicate these numbers in our TCK status updates? Perhaps we could have a script that automates collecting them, so anyone can copy/paste them easily into
an email or even create another Jenkins job that performs release validation.
Running these tests, in the same way we did for EE 8, on each stand-alone TCK could be a useful measure or our progress. (Also, removing any tests that are pruned might also be good.)
+1. We will remove the pruned tests and have tracking issues for that.
Currently all the porting kit references are "undefined."
What happens after the porting key references are "defined?" Does that help GlassFish 6.0 to run the Standalone TCKs?
Many of the sub-projects as well. Guru has started working on BV, DI and CDI porting kits, but has not progressed much past basic build problems with GlassFish 6. I'm trying to figure out what we can to do unblock that but it's slow going.
Is this captured by a Jakarta Platform level issue?
If the API teams can create their own Jenkins jobs, performing the tests to their requirements that would be even more data and would be even more useful. They should have jobs they used from EE 8 and they can evolve from those.
We can decide if any of this will be a goal for the forthcoming Milestone release, for example we could say -- it is a goal to generate the .zip files with their planned target name and maybe a stretch goal to have a test result listed in the Jenkins job
results.
Maybe we would like a goal about the other TCKs -- for example, just that we're running them against GlassFish (saying nothing about a result goal, for example).
Lastly, I'll just comment that the 'nightly' directory is quite full of lots of files. Some from EE 8, some from EE 9. I'd recommend we consider moving all the
EE 9 TCKs into an EE9 sub-folder (or maybe someone wants to suggest an alternative organization they like better).
What do you and others think?
Thank you for these ideas and communicating them on this mailing list!
Speaking of ideas, Cesar Hernandez raised the idea of doing a video or blog, to encourage more community involvement on the Platform TCK. I would very much like to do this soon, if others are willing to contribute. I think that we could
start a new email thread to discuss possible topics and what people would like to learn more about, including the history of the TCKs.
Also, following up on another idea from the discussion on branching, IMO we should document our process around the Platform TCKs, for the "prospective maintainers, years from now". This is something that is started on https://github.com/eclipse-ee4j/jakartaee-tck/wiki
but IMO could use more content and updates.
-- Ed
On 5/19/2020 11:48 AM, Alwin Joseph wrote:
Hi Scott,
Yes that is correct. We build and run the standalone TCKs whose source is jakartaee-tck repo using [1]. This job uses the scripts under /docker to build & run standalone TCKs.
-build_standalonetck.sh to build all the standalone TCKs(takes tck name as parameter)
-{tckname}tck.sh to run the standalone TCK.
But some of the api projects have their own job for building and running these standalone TCKs in its own CI instance by cloning the jakartaee-tck repo. For eg: The jobs in
https://ci.eclipse.org/jsonp/,
https://ci.eclipse.org/jaxrs/ were used to build and run jsonp & jaxrs standalone TCKs.
Activation & mail tcks are built & run in their own CI instances - [2] & [3] respectively.
Regards,
Alwin
On 19/05/20 11:35 PM, Scott Marlow wrote:
All,
I wasn't quite sure how the Jakarta Platform Stand-alone TCKs were being built, but think that I found the answer via the standalonetck-nightly-build-run-master [1] job, which runs on a weekly basis (last run was on May 13). It takes just
under two hours to run the standalonetck-nightly-build-run-master [1].
The job writes to the jakartaee-tck/master/nightly downloads [2] folder, note that the Eclipse Jakarta EE 8 platform files are still there as well.
_______________________________________________
jakartaee-tck-dev mailing list
jakartaee-tck-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
_______________________________________________
jakartaee-tck-dev mailing list
jakartaee-tck-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
|
