[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
Re: [jakarta.ee-spec] [External] : Re: Process for TCK service releases that include TCK updates for running signature tests on newer JDK versions...
|
On 8/31/21 12:14 PM, Ed Bratt wrote:
Hmm. I'm probably diverting this away from the original subject
-- I suspect this might be more useful in a general 'TCK release
process requirements' thread.
I think that makes sense. In general, whomever is taking
responsibility to handle any update (via Challenge, JDK update,
or just ongoing evolution) ought to have the facilities to
validate, whatever their change might be -- before making any
new release. Adding that as an explicit requirement for all TCK
bug-fix updates, seems appropriate.
This will/could become a bit more problematic as we start to
expand the number of TCKs that are completely independent of the
Platform TCK. I suspect that most component APIs do not consider
testing with a Platform CI as part of their requirement set.
With the centralized Platform TCK project, most of this testing
was built into their modify-build-validate process. If a change
were made to RESTFul Web Services, in EE 9 or earlier, that
change would be validated in both the stand-alone and platform
TCKs as part of their standard build/release process. When
RESTful web services provides an update to their TCK (in EE 10
and beyond), should we expect that team to have validated their
TCK change on both their own component compatible implementation
AND a platform implementation?
RESTFul Web Services is a `Wave 4` SPEC, meaning that the RESTFul
Web Services SPEC ballot will be well ahead of the Platform SPEC
ballot. In previous releases, the RESTFul Web Services Platform
TCK tests were not frozen until the Platform TCK was completed but
for EE10 we will have some Platform tests in the RESTFul Web
Services SPEC TCK. Can we validate SPEC level tests at the time
of Ballot (e.g. only non-platform RESTFul Web Services TCK tests
are validated at ballot time), and allow SPEC teams to update
their TCKs after their initial ballot completes?
From a development perspective, I think that SPEC teams will need
to be able to update their (Platform level) TCK tests after their
ballot completes as the platform requirements could change or more
simply test bugs may need to be fixed, perhaps in the form of a
RESTFul Web Services TCK service release that only releases the
TCK artifacts from the RESTFul Web Services 3.1 branch.
Perhaps the Platform TCK project will need to include some
tooling that allows for the test running of component TCK
changes. I realize this is a bit problematic. An alternative
would be to simply put that onus onto the component Spec. dev.
teams. If we provided enough boiler-plate to make it easy for
them to run these tests, perhaps this isn't too much of an
additional burden.
-- Ed
On 8/31/2021 7:59 AM, Scott Marlow
wrote:
From your point, I am thinking that the TCK producing team
needs to block releasing of the TCK until verification has
been provided that the TCK can be passed (to be defined as to
how). I also want to say that all compatible implementations
need to be treated the same and IMO our process shouldn't
dictate that https://ci.eclipse.org/jakartaee-tck
must be used to verify the produced TCK.
Scott
On 8/27/21 5:13 PM, Ed Bratt wrote:
Currently, it is possible to update the parameters that
runs tests using GlassFish so that it can pull down a
preliminary TCK and runs the tests. GlassFish has the
benefit that it can be used this way for all TCK tests.
Other implementations may not be as flexible, but there must
be a way for the TCK producing team to test the TCK product
it's going to produce. While this tooling may be unique
across the implementation landscape -- I am imagining that
some kind of test set up will be implemented, regardless
which vendor provides the initial ballot compatible
implementation. There is simply no way we can produce a TCK
and not have a process in place that provides for modify,
built, test, review-results, modify, build, test ... cycles.
This will have to be part of the process for any candidate
compatible implementation that we intend to use for any
release ballot.
Probably this needs to be discussed in more detail. These
requirements probably need to be made more explicit and the
component teams will probably need to become more aware of
these obligations. (i.e. they won't be able to just assume
that GlassFish will be primed and ready when they think
they're ready to deliver their Spec/API/TCK)
We can't release TCKs without confirming that they run in
the anticipated environment and they produce the anticipated
compatibility verification. I don't see this as much of a
problem for just excluding tests but if we add a feature
(even if it's just a new Java SE version), we have to be
able to test and iterate this prospective update with some
compatible version. We will simply have to include this
obligation on at least one of the initial ballot compatible
implementations. It's got to be a two way partnership -- not
a unilateral relationship.
-- Ed
On 8/27/2021 1:01 PM, Scott
Marlow wrote:
On 8/26/21 8:24 PM, Ed Bratt
wrote:
I would recommend this be brought to the Specification
Committee for discussion and once everyone is satisfied,
that a resolution be proposed to approve this as a new
TCK process guide.
It would be nice to see a change-bar version of the
document (I guess I can get that from the current source
diff)
Under the section 'Process for Releasing a point
revision' (the last section) -- My preference would be
to include documentation to reference a compatible
implementation that successfully passed the revised TCK.
For exclude only updates, this should be easy if you can
get one or more of the original compatible
implementations to rerun their tests. For updates, that
add new Java versions, this could be more difficult but,
presumably we'd be releasing the update for the purposes
of qualifying a particular implementation so, probably
that version could be included (though I guess that
might not be an open-source compatible implementation).
In my opinion, we always want evidence that the TCK was
run and an implementation successfully passed it.
Referencing the certification request associated with
that implementation would be the easiest way to capture
this.
The particular implementation that created the TCK
challenge hasn't yet created their certification request
yet as they are blocked on waiting for the new TCK release
to be published. They may also be waiting for other TCK
challenges to be processed before creating their
compatibility request. I do like the suggestion but I'm
not yet understanding how we can accomplish it. At the
very least, I would like the particular implementation to
download the (not yet released) TCK to verify it after it
has been built and communicate that the (not yet released)
TCK is working as expected.
We are also transitioning over to not having a reference
implementation to use for verifying not yet released
TCKs. We can ask the various compatible implementations
to test the new TCK but we cannot expect them to do that
in a timely manner.
For reference, the referenced section currently contains:
"
Process for releasing a point revision of a TCK
The process for releasing a point revision of a TCK
entails filing an issue in the jakartaee/specifications
repository with the following details:
- Link to the TCK release to be published.
- Updated TCK links in the specification's _index.md
file.
- A high-level description of what tests were excluded
from the TCK and why.
"
Scott
On 8/18/2021 9:16 AM, Scott
Marlow wrote:
On 8/5/21 11:01 AM, Kevin
Sutter wrote:
Hi
Scott,
I
think we should pursue an update to the TCK process
to allow service releases to fix Signature tests
related to newer versions of Java. Not sure if we
have to be that specific, but we do need to allow
for these type of updates. The alternative of
ignoring certain tests gets tricky and nebulous
since ignored tests may actually have an issue, but
we wouldn't know as casual observers. It would be
much better to be clearer and service releases would
allow us to be clear. Thanks!
I just updated https://github.com/jakartaee/jakarta.ee/pull/1018
to be less specific about service releases to fix
tests for newer versions of Java (could be signature
test changes or dealing with removal of Java security
manager or something else).
Does anyone disagree with updating the TCK Process
version from `1.0` to `1.1`? For what
reason/condition should we update the version to
`2.0`?
Does anyone else need to review https://github.com/jakartaee/jakarta.ee/pull/1018
before it gets merged?
---------------------------------------------------
Kevin Sutter
STSM, Jakarta EE and MicroProfile architect @ IBM
e-mail: sutter@xxxxxxxxxx
Twitter: @kwsutter
phone: tl-553-3620 (office), 507-253-3620 (office)
LinkedIn: https://www.linkedin.com/in/kevinwsutter
Part-time schedule: Tue, Wed, Thu (off on Mon and
Fri)
From:
"Scott Marlow"
<smarlow@xxxxxxxxxx>
To:
jakarta.ee-spec@xxxxxxxxxxx
Date:
08/05/2021
08:25
Subject:
[EXTERNAL]
[jakarta.ee-spec] Process for TCK service releases
that include TCK updates for running signature tests
on newer JDK versions...
Sent
by: "jakarta.ee-spec"
<jakarta.ee-spec-bounces@xxxxxxxxxxx>
For Jakarta EE Platform
9.1+ supports implementations running TCK
compatibility certification tests on JDK versions
Java SE 8, Java SE 11+. In support of running TCK
tests on JDK versions greater than SE 11, we expect
that some tests will need to be revised (e.g. see
jaxb-tck/issues/57 [1] for updating signature tests
related to need new signature tooling library and
signature map files).
Last December, we
started making changes to the `TCK Process 1.0` that
includes the following text [2] which introduces an
alternative to excluding (challenged) TCK tests:
`The specification
project may approve (user) workarounds for an
`accepted` TCK challenge (as alternative to
excluding TCK tests).`
My question today is
whether the above quoted text is enough to cover
Jakarta EE 9.1 compatibility certification requests
against Java SE 17 (which will include signature
test failures due to jaxb-tck/issues/57 [1])? If
the answer/vote is yes, certain signature test
failures can be ignored on newer JDK versions, if
and only if the signature test failure is caused by
inadequate TCK signature support for the relevant
Java SE (e.g. JDK 17) version. If the answer/vote
is no, we will need an additional TCK process change
to specifically allow a SPEC TCK service release
that updates signature tests to resolve the
signature test failure (e.g. allow
jakarta-xml-binding-tck-3.0.2.zip [4] to be
officially released by Spec team so that
implementations can submit certification requests
against jakarta-xml-binding-tck-3.0.2.zip).
Scott
[1] https://github.com/eclipse-ee4j/jaxb-tck/issues/57
[2] https://github.com/jakartaee/jakarta.ee/pull/1018/files#diff-1fe254a18287c0db31fd9cb0a6bca11b1efda926095c3a65b73ef2ae0c89360dR223
[3] https://jakarta.ee/committees/specification/tckprocess/
[4] https://download.eclipse.org/ee4j/jakartaee-tck/jakartaee9-eftl/promoted/jakarta-xml-binding-tck-3.0.2.zip_______________________________________________
jakarta.ee-spec mailing list
jakarta.ee-spec@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec
_______________________________________________
jakarta.ee-spec mailing list
jakarta.ee-spec@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec
_______________________________________________
jakarta.ee-spec mailing list
jakarta.ee-spec@xxxxxxxxxxx
To unsubscribe from this list, visit https://urldefense.com/v3/__https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec__;!!ACWV5N9M2RV99hQ!awIHuBnS0vFrSdaLJF1CkeydaJ6HBDuZO-HU31C_-GlXemvLd-yxK7aypDQTQqQ$