[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [jakartaee-tck-dev] Weekly building of Jakarta Platform Stand-alone TCKs...
|
(Oye, this just keeps getting longer. Sorry ...)
Yes, Kudos to Mark Thomas for his work on Web Socket TCK. Scott,
you ask if we should wait for the Platform TCK before running the
stand-alone TCKs -- I guess that is a question to consider with
the API project committer teams. If they are following the pattern
that Mark is using, they will likely want results sooner. But,
it's also a resource issue (when can someone in the TCK project
actually make the changes needed?).
I raised the question about production of these stand-alone TCKs
because -- if one of the Milestone 1 goals is to have TCKs from
each API, this action (produce each of these out of the
jakartaee-tck project) will need to be completed. I do not know if
there is sufficient time and resource to accomplish this in time
for a June 23 milestone. There are 21 specification TCKs that are
produced out of this project. If each of these is a goal, then the
weight will lie heavily with this committer team.
Those folder views at download.eclipse.org are part of the
download infrastructure that Eclipse provides. See more here.
There is reference to a p2.index file which may allow additional
and/or customized file viewing. I don't know any more than that
this is referenced on the Eclipse IT feature wiki. Long ago, we
put the date string into the file name and kept an updated link to
the 'latest' copy of each build. Anyone that wanted to track the
current build could consistently pull the 'latest' file. Specific
builds (within some upper bound) could be found in the directory.
Someone could take this task and decide what to do and implement
it, if that's what the committer team wants.
Almost certainly most files in the nightly folder, even the older
version files could be removed or moved elsewhere. All of the
necessary files should have been copied (or released) to their
final release folder when they were promoted. Given that we may
need to deal with maintenance, whatever organization we decide on,
I would recommend we include a version specific qualifier. If we
choose to remove legacy files actively, I'm sure the Eclipse
administrators will be more happy with us since this is a shared
project resource.
Steve, in addition to what you wrote, details about the Jenkins
TCK jobs are kept on this
wiki. It may need to evolve so if anyone spots errors,
hopefully they can just make updates.
Yes, GlassFish runs both the Platform TCK (AKA CTS) AND each of
the stand-alone TCKs are run against it. We conclude it is
compatible when all these tests pass. Some external TCKs are run
with porting kit packages. Some use other procedures to run. The
rules, prescriptions, procedures, etc. should be documented in
each TCK users guide. We have tried with varying success to have
the API teams take more responsibility defining and implementing
the process and procedures required to run their TCKs against,
whatever would qualify as a compatible implementation. Mark's
work, referenced above is exemplary! In some cases, the
stand-alone TCK is run against a fully delivered Java application.
In other cases, the compatible implementation needs to run within
something else (frequently, an Application Server but not always).
The "generic" GlassFish stand-alone TCK test runner, in many cases
just uses GlassFish in some ways, as if it was a delivery archive.
The scripts pull out the necessary JAR files, then start the test
as agreed upon between the TCK test developer and the API project
team and documented in the TCK Users Guide. The results are
accumulated (at least for EE 8 example, here).
In some cases, it makes sense to run these tests against a
packaged application. In others, it probably doesn't (though the
API team can certainly produce something and/or work with an
implementation team, if it desires). There is probably room for
improvement, but likely this could be considered at a later time.
More details may be found about the Standalone TCK Build commands,
here.
At the end of the release, the ballot submission will contain a
reference to the TCK and whatever run-time components were used to
generate the result, demonstrating that a compatible
implementation exists and it passes the TCK. One needs to look at
each one of the Ballot record pages, to determine what was used
for the ballots. I think there are differing opinions about the
requirements for when it is required to run an imported component
against the stand-alone TCK -- and also when those results should
(or must) be reported. I suspect there isn't a lot of urgency to
clarify this further than it is already.
Here is the wiki describing the
TCK jobs (same link as above). Some sample jobs run Jakarta
EE Platform TCK (AKA CTS)
Some sample jobs run Jakarta EE stand-alone TCKs (those derived
from the jakartaee-tck project)
Both of these test runners require a GlassFish distribution.
Users can kick off these test runs, via job parameters, set the
list of TCKs (or test suites from Platform TCK) to run and point
them to any GlassFish build. It is my belief that any component
that is built and released should pass that that components
stand-alone TCK. So, GlassFish should pass the Platform TCK and
the stand-alone TCKs for each and every required Jakarta EE
component that it includes (it includes all optional components).
The Specification (in this case, both the written specification
and the TCK Users Guide) should define what is required.
Many API projects have their own Jenkins TCK test runners.
Ultimately, all of these will need to be updated for EE 9.
Obviously, we can leave the task of Jenkins TCK jobs in other
projects to those projects. Eventually, it would be nice to try
and optimize these so that we aren't running jobs any more often
than needed. Over testing (e.g. building on calendar events vs.
repository update triggers) can become an issue when we close in
on releases and all the projects are trying to finalized and
complete their last changes at the same time -- and the Jenkins CI
infrastructure is shared. Some of this is worked out in GlassFish
CI infrastructure, but there is certainly more to do here and many
of the dependency relationships are less than direct.
Scott asks if GlassFish relies on running stand-alone TCK nightly
-- I am not certain how you are using rely here. GlassFish has
it's own qualification tests to determine if each build job
succeeds or fails. I am not aware of a built-in requirement for
running any formal TCK suite leading to success or failure of a
build. There are "quick-look" tests which do include some TCK
related functionality, but TCK results are generally produced
independently. Certainly, for GlassFish, this has been determined
to be way to time-consuming since we generally don't even like
waiting an hour for build qualification runs. Before migrating to
Eclipse, our goal was never to produce releases that did not pass
the TCKs. We did not always achieve that goal and during an active
development phase of one or more components, the result of the TCK
is potentially un(under?)defined. I suspect it's up to the broader
EE4J community to decide -- maybe just the subset that works on
the implementations? -- to decide what dependency qualifications
are required at each build level (nightly, promoted, released).
Scott asks about Porting Kit issue references -- Porting kits are
the mechanism that the process defines for adapting an additional
TCK to an Application Server. The porting kits are specific to an
Application Server. So, the kits I reference below, will produce
TCK compatibility results for those specifications (BV, CDI, DI),
for Eclipse GlassFish. I would say that the issues to track that
work belong with the GlassFish project, which is where I created
them. In my view, the Platform level is concerned with: Compatible
Implementations and their TCK results at the time of the ballot.
The Platform committer team certainly cares about the Platform TCK
and any ancillary requirements for that Platform specification so
-- they will care that an Application Server compatibility
application contains all the necessary TCK results, obtained in
whatever required fashion is specified. So, in this regard, the
Platform Committer team needs to see that GlassFish contains the
required results. Bugs/Evolution/Etc. of those specific porting
kits are probably the implementation level scope. The platform
team certainly cares that each component team delivers a TCK and
produces a compatible implementation result. I think it's up to
each implementation if they need to use a porting kit, or if they
can achieve the required results directly. (Maybe someone else
could confirm or correct the porting kit specific requirements.)
Absolutely I commend the idea of soliciting more involvement, in
any way we can. We should also consider promoting the idea that
the component specification committer teams or team members need
to start working more continuously with their TCKs and possibly
addressing some of the commentary and/or questions I speculate
about, above.
Probably some of what is here, should find its way to the JakartaEE-TCK
Wiki. We have tried to create lots of documentation about
the TCKs. These materials need to be revised along with the TCKs
themselves (and the environment and the process, etc. etc.) Please
help keep it revised and current. I'm hoping nothing here is in
conflict with the wikis or process. If there are conflicts, I'd
recommend you go with the prior descriptions and requirements.
Just to summarize some potential reporting metrics:
- Platform TCK is produced
- Platform TCK runs against Eclipse GlassFish and produces a
result (any result)
- Platform TCK runs against Eclipse GlassFish Web Profile and
produces a result
- Platform Committer team evaluates and approves all platform
requirements are described and are met
- Platform compatible implementation (and probably Platform
Spec) are ready for ballot
- Component TCKs are each produced
- The build jobs succeed and generate a TCK Zip
- The generated TCK Zip is against Eclipse GlassFish and
produces a test result
- Test result passes
- Component TCK team reports passing result against their
compatible implementation and approves all component
requirements are described and met
- Component compatible implementation and component
specification are ready for ballot
I haven't heard any concerns about setting a goal to generate the
Platform TCK for the milestone. I would presume there's no
argument against a goal to run it against Eclipse GlassFish --
perhaps reporting those results, perhaps not. The additional
bullets (and maybe some I don't have here) should be settled on so
we can all be clear what our goals are. (I have heard that other
implementation teams are working with the TCKs -- it might be nice
if we could get them to share their goals and/or desires from this
project might be, but I don't know that we can count on any of
that since we need to produce these before anyone will want to
make any statements about using them.)
Alternatively, we could just say -- here is the
road-map/continuum. We will get as far as we get and just report
where we are as we progress to the Milestone 1 date. That might be
okay.
Whatever we settle on, we should try to be clear when we are
reporting this to the other teams/committees so they can set their
expectations appropriately.
-- Ed
On 5/20/2020 3:08 AM, Steve Millidge
(Payara) wrote:
I’ve
raised an issue on the GlassFish project to get our Jenkins
master build job to upload an artefact to the Eclipse
download site. See https://github.com/eclipse-ee4j/glassfish/issues/23022
I’ll update this list when that issue is done and we have a
download there.
AFAIK GlassFish uses these jobs to run the
CTS
https://ci.eclipse.org/glassfish/view/cts/ But Ed/Dmitry
and their team set those jobs up so will be more familiar.
Steve
Thanks Ed, good to raise validation
(and other things) for discussion here! :) More inline
below...
Some additional comments and ideas:
I think that some of the TCK build paths are failing
-- or, what I'm actually observing is that some TCKs
are not at the same version as their specification
(i.e. annotations-tck is either not updated, or the
build job is still creating files with the name
'1.3.0' in their string. I presume the Annotations TCK
will become 2.x.x when it's updated. Some of the TCK
zip files are updated. WebSocket, for example is 2.0.0
so I believe that is actually getting some attention).
We may find that some of these build paths are failing
along the way and simply not generating a zip.
A useful metric could be simply that these zip files
are being produced and have the correct version IDs.
(By a count I did last week, there were 6 zip files
that seemed to be updated using this metric. There
could be more this week, I haven't checked yet.)
In addition when you look at the Jenkins job under
JUnit Reports (e.g. here),
you will see a few TCK entry lines (12 right now) --
nearly all are failing.
These test jobs use GlassFish builds and will perform
a validation on each stand-alone TCK. I hesitate to
call that compatibility validation, because there was
concern in EE 8 about how those tests are run -- but
it is a potential comparative point.
Does the GlassFish project also run
each stand-alone TCK, as well? Or rely on the https://ci.eclipse.org/jakartaee-tck/job/standalonetck-nightly-build-run-master for
validation? If we do rely
on standalonetck-nightly-build-run-master, one low cost
improvement, is to have TCK failures trigger email
notification to be sent to those interested (or we could
have a new public mailing list that TCK failures are
sent to, if others would join such a potentially noisy
mailing list :). We currently do this for the nightly
build of the Platform TCK.
Regardless, there were
23 entries in EE 8, just under 19K tests that
passed. These numbers will change for EE 9 due to
pruning. But we can track this evolution as the work
progresses.
Should we wait until we are running
with a daily GlassFish master (6.0) build to communicate
these numbers in our TCK status updates? Perhaps we
could have a script that automates collecting them, so
anyone can copy/paste them easily into an email or even
create another Jenkins job that performs release
validation.
Running these tests, in the same way we did for EE 8,
on each stand-alone TCK could be a useful measure or
our progress. (Also, removing any tests that are
pruned might also be good.)
+1. We will remove the pruned tests
and have tracking issues for that.
Currently all the porting kit references are
"undefined."
What happens after the porting key
references are "defined?" Does that help GlassFish 6.0
to run the Standalone TCKs?
Many of the sub-projects as well. Guru has started
working on BV, DI and CDI porting kits, but has not
progressed much past basic build problems with
GlassFish 6. I'm trying to figure out what we can to
do unblock that but it's slow going.
Is this captured by a Jakarta
Platform level issue?
If the API teams can create their own Jenkins jobs,
performing the tests to their requirements that would
be even more data and would be even more useful. They
should have jobs they used from EE 8 and they can
evolve from those.
We can decide if any of this will be a goal for the
forthcoming Milestone release, for example we could
say -- it is a goal to generate the .zip files with
their planned target name and maybe a stretch goal to
have a test result listed in the Jenkins job results.
Maybe we would like a goal about the other TCKs --
for example, just that we're running them against
GlassFish (saying nothing about a result goal, for
example).
Lastly, I'll just comment that the 'nightly'
directory is quite full of lots of files. Some from EE
8, some from EE 9. I'd recommend we consider moving
all the EE 9 TCKs into an EE9 sub-folder (or maybe
someone wants to suggest an alternative organization
they like better).
What do you and others think?
Thank you for these ideas and
communicating them on this mailing list!
Speaking of ideas, Cesar Hernandez
raised the idea of doing a video or blog, to encourage
more community involvement on the Platform TCK. I would
very much like to do this soon, if others are willing to
contribute. I think that we could start a new email
thread to discuss possible topics and what people would
like to learn more about, including the history of the
TCKs.
Also, following up on another idea
from the discussion on branching, IMO we should document
our process around the Platform TCKs, for the
"prospective maintainers, years from now". This is
something that is started on https://github.com/eclipse-ee4j/jakartaee-tck/wiki
but IMO could use more content and updates.
-- Ed
On 5/19/2020 11:48 AM, Alwin
Joseph wrote:
Hi Scott,
Yes that is correct. We build and run the
standalone TCKs whose source is jakartaee-tck repo
using [1]. This job uses the scripts under /docker
to build & run standalone TCKs.
-build_standalonetck.sh to build all the standalone
TCKs(takes tck name as parameter)
-{tckname}tck.sh to run the standalone TCK.
But some of the api projects have their own job for
building and running these standalone TCKs in its
own CI instance by cloning the jakartaee-tck repo.
For eg: The jobs in https://ci.eclipse.org/jsonp/,
https://ci.eclipse.org/jaxrs/
were used to build and run jsonp & jaxrs
standalone TCKs.
Activation & mail tcks are built & run in
their own CI instances - [2] & [3] respectively.
Regards,
Alwin
On 19/05/20 11:35 PM, Scott
Marlow wrote:
All,
I wasn't quite sure how
the Jakarta Platform Stand-alone TCKs were
being built, but think that I found the
answer via the
standalonetck-nightly-build-run-master [1]
job, which runs on a weekly basis (last run
was on May 13). It takes just under two
hours to run
the standalonetck-nightly-build-run-master
[1].
The job writes to
the jakartaee-tck/master/nightly downloads
[2] folder, note that the Eclipse Jakarta EE
8 platform files are still there as well.
_______________________________________________
jakartaee-tck-dev mailing list
jakartaee-tck-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
_______________________________________________
jakartaee-tck-dev mailing list
jakartaee-tck-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
_______________________________________________
jakartaee-tck-dev mailing list
jakartaee-tck-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
