| Re: [jakartaee-tck-dev] Weekly building of Jakarta Platform Stand-alone TCKs... |
Some additional comments and ideas:
I think that some of the TCK build paths are failing -- or, what I'm actually observing is that some TCKs are not at the same version as their specification (i.e. annotations-tck is either not updated, or the build job is still creating files with the name '1.3.0' in their string. I presume the Annotations TCK will become 2.x.x when it's updated. Some of the TCK zip files are updated. WebSocket, for example is 2.0.0 so I believe that is actually getting some attention). We may find that some of these build paths are failing along the way and simply not generating a zip.
A useful metric could be simply that these zip files are being produced and have the correct version IDs. (By a count I did last week, there were 6 zip files that seemed to be updated using this metric. There could be more this week, I haven't checked yet.)
In addition when you look at the Jenkins job under JUnit Reports (e.g. here), you will see a few TCK entry lines (12 right now) -- nearly all are failing.
These test jobs use GlassFish builds and will perform a validation on each stand-alone TCK. I hesitate to call that compatibility validation, because there was concern in EE 8 about how those tests are run -- but it is a potential comparative point.
Regardless, there were 23 entries in EE 8, just under 19K tests that passed. These numbers will change for EE 9 due to pruning. But we can track this evolution as the work progresses.
Running these tests, in the same way we did for EE 8, on each stand-alone TCK could be a useful measure or our progress. (Also, removing any tests that are pruned might also be good.)
Currently all the porting kit references are "undefined."
Many of the sub-projects as well. Guru has started working on BV, DI and CDI porting kits, but has not progressed much past basic build problems with GlassFish 6. I'm trying to figure out what we can to do unblock that but it's slow going.
If the API teams can create their own Jenkins jobs, performing the tests to their requirements that would be even more data and would be even more useful. They should have jobs they used from EE 8 and they can evolve from those.
We can decide if any of this will be a goal for the forthcoming Milestone release, for example we could say -- it is a goal to generate the .zip files with their planned target name and maybe a stretch goal to have a test result listed in the Jenkins job results.
Maybe we would like a goal about the other TCKs -- for example, just that we're running them against GlassFish (saying nothing about a result goal, for example).
Lastly, I'll just comment that the 'nightly' directory is quite full of lots of files. Some from EE 8, some from EE 9. I'd recommend we consider moving all the EE 9 TCKs into an EE9 sub-folder (or maybe someone wants to suggest an alternative organization they like better).
What do you and others think?
-- Ed
On 5/19/2020 11:48 AM, Alwin Joseph wrote:
Hi Scott,
Yes that is correct. We build and run the standalone TCKs whose source is jakartaee-tck repo using [1]. This job uses the scripts under /docker to build & run standalone TCKs.
Regards,
-build_standalonetck.sh to build all the standalone TCKs(takes tck name as parameter)
-{tckname}tck.sh to run the standalone TCK.
But some of the api projects have their own job for building and running these standalone TCKs in its own CI instance by cloning the jakartaee-tck repo. For eg: The jobs in https://ci.eclipse.org/jsonp/, https://ci.eclipse.org/jaxrs/ were used to build and run jsonp & jaxrs standalone TCKs.
Activation & mail tcks are built & run in their own CI instances - [2] & [3] respectively.
Alwin
[1] https://ci.eclipse.org/jakartaee-tck/job/standalonetck-nightly-build-run-master/
[2] https://ci.eclipse.org/jaf/
[3] https://ci.eclipse.org/mail/
On 19/05/20 11:35 PM, Scott Marlow wrote:
All,
I wasn't quite sure how the Jakarta Platform Stand-alone TCKs were being built, but think that I found the answer via the standalonetck-nightly-build-run-master [1] job, which runs on a weekly basis (last run was on May 13). It takes just under two hours to run the standalonetck-nightly-build-run-master [1].
The job writes to the jakartaee-tck/master/nightly downloads [2] folder, note that the Eclipse Jakarta EE 8 platform files are still there as well.
Scott
_______________________________________________ jakartaee-tck-dev mailing list jakartaee-tck-dev@xxxxxxxxxxx To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
_______________________________________________ jakartaee-tck-dev mailing list jakartaee-tck-dev@xxxxxxxxxxx To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev