[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
Re: [jakarta.ee-spec] [External] : Re: Process for TCK service releases that include TCK updates for running signature tests on newer JDK versions...
|
Hmm. I'm probably diverting this away from the original subject
-- I suspect this might be more useful in a general 'TCK release
process requirements' thread.
I think that makes sense. In general, whomever is taking
responsibility to handle any update (via Challenge, JDK update, or
just ongoing evolution) ought to have the facilities to validate,
whatever their change might be -- before making any new release.
Adding that as an explicit requirement for all TCK bug-fix
updates, seems appropriate.
This will/could become a bit more problematic as we start to
expand the number of TCKs that are completely independent of the
Platform TCK. I suspect that most component APIs do not consider
testing with a Platform CI as part of their requirement set. With
the centralized Platform TCK project, most of this testing was
built into their modify-build-validate process. If a change were
made to RESTFul Web Services, in EE 9 or earlier, that change
would be validated in both the stand-alone and platform TCKs as
part of their standard build/release process. When RESTful web
services provides an update to their TCK (in EE 10 and beyond),
should we expect that team to have validated their TCK change on
both their own component compatible implementation AND a platform
implementation? Perhaps the Platform TCK project will need to
include some tooling that allows for the test running of component
TCK changes. I realize this is a bit problematic. An alternative
would be to simply put that onus onto the component Spec. dev.
teams. If we provided enough boiler-plate to make it easy for them
to run these tests, perhaps this isn't too much of an additional
burden.
-- Ed
On 8/31/2021 7:59 AM, Scott Marlow
wrote:
From your point, I am thinking that the TCK producing team
needs to block releasing of the TCK until verification has been
provided that the TCK can be passed (to be defined as to how).
I also want to say that all compatible implementations need to
be treated the same and IMO our process shouldn't dictate that https://ci.eclipse.org/jakartaee-tck
must be used to verify the produced TCK.
Scott
On 8/27/21 5:13 PM, Ed Bratt wrote:
Currently, it is possible to update the parameters that runs
tests using GlassFish so that it can pull down a preliminary
TCK and runs the tests. GlassFish has the benefit that it can
be used this way for all TCK tests. Other implementations may
not be as flexible, but there must be a way for the TCK
producing team to test the TCK product it's going to produce.
While this tooling may be unique across the implementation
landscape -- I am imagining that some kind of test set up will
be implemented, regardless which vendor provides the initial
ballot compatible implementation. There is simply no way we
can produce a TCK and not have a process in place that
provides for modify, built, test, review-results, modify,
build, test ... cycles. This will have to be part of the
process for any candidate compatible implementation that we
intend to use for any release ballot.
Probably this needs to be discussed in more detail. These
requirements probably need to be made more explicit and the
component teams will probably need to become more aware of
these obligations. (i.e. they won't be able to just assume
that GlassFish will be primed and ready when they think
they're ready to deliver their Spec/API/TCK)
We can't release TCKs without confirming that they run in the
anticipated environment and they produce the anticipated
compatibility verification. I don't see this as much of a
problem for just excluding tests but if we add a feature (even
if it's just a new Java SE version), we have to be able to
test and iterate this prospective update with some compatible
version. We will simply have to include this obligation on at
least one of the initial ballot compatible implementations.
It's got to be a two way partnership -- not a unilateral
relationship.
-- Ed
On 8/27/2021 1:01 PM, Scott Marlow
wrote:
On 8/26/21 8:24 PM, Ed Bratt
wrote:
I would recommend this be brought to the Specification
Committee for discussion and once everyone is satisfied,
that a resolution be proposed to approve this as a new TCK
process guide.
It would be nice to see a change-bar version of the
document (I guess I can get that from the current source
diff)
Under the section 'Process for Releasing a point
revision' (the last section) -- My preference would be to
include documentation to reference a compatible
implementation that successfully passed the revised TCK.
For exclude only updates, this should be easy if you can
get one or more of the original compatible implementations
to rerun their tests. For updates, that add new Java
versions, this could be more difficult but, presumably
we'd be releasing the update for the purposes of
qualifying a particular implementation so, probably that
version could be included (though I guess that might not
be an open-source compatible implementation). In my
opinion, we always want evidence that the TCK was run and
an implementation successfully passed it. Referencing the
certification request associated with that implementation
would be the easiest way to capture this.
The particular implementation that created the TCK
challenge hasn't yet created their certification request yet
as they are blocked on waiting for the new TCK release to be
published. They may also be waiting for other TCK
challenges to be processed before creating their
compatibility request. I do like the suggestion but I'm not
yet understanding how we can accomplish it. At the very
least, I would like the particular implementation to
download the (not yet released) TCK to verify it after it
has been built and communicate that the (not yet released)
TCK is working as expected.
We are also transitioning over to not having a reference
implementation to use for verifying not yet released TCKs.
We can ask the various compatible implementations to test
the new TCK but we cannot expect them to do that in a timely
manner.
For reference, the referenced section currently contains:
"
Process for releasing a point revision of a TCK
The process for releasing a point revision of a TCK entails
filing an issue in the jakartaee/specifications repository
with the following details:
- Link to the TCK release to be published.
- Updated TCK links in the specification's _index.md file.
- A high-level description of what tests were excluded
from the TCK and why.
"
Scott
On 8/18/2021 9:16 AM, Scott
Marlow wrote:
On 8/5/21 11:01 AM, Kevin
Sutter wrote:
Hi
Scott,
I
think we should pursue an update to the TCK process to
allow service releases to fix Signature tests related
to newer versions of Java. Not sure if we have to be
that specific, but we do need to allow for these type
of updates. The alternative of ignoring certain tests
gets tricky and nebulous since ignored tests may
actually have an issue, but we wouldn't know as casual
observers. It would be much better to be clearer and
service releases would allow us to be clear. Thanks!
I just updated https://github.com/jakartaee/jakarta.ee/pull/1018
to be less specific about service releases to fix tests
for newer versions of Java (could be signature test
changes or dealing with removal of Java security manager
or something else).
Does anyone disagree with updating the TCK Process
version from `1.0` to `1.1`? For what reason/condition
should we update the version to `2.0`?
Does anyone else need to review https://github.com/jakartaee/jakarta.ee/pull/1018
before it gets merged?
---------------------------------------------------
Kevin Sutter
STSM, Jakarta EE and MicroProfile architect @ IBM
e-mail: sutter@xxxxxxxxxx
Twitter: @kwsutter
phone: tl-553-3620 (office), 507-253-3620 (office)
LinkedIn: https://www.linkedin.com/in/kevinwsutter
Part-time schedule: Tue, Wed, Thu (off on Mon and Fri)
From:
"Scott Marlow" <smarlow@xxxxxxxxxx>
To:
jakarta.ee-spec@xxxxxxxxxxx
Date:
08/05/2021 08:25
Subject:
[EXTERNAL]
[jakarta.ee-spec] Process for TCK service releases
that include TCK updates for running signature tests
on newer JDK versions...
Sent
by: "jakarta.ee-spec"
<jakarta.ee-spec-bounces@xxxxxxxxxxx>
For Jakarta EE Platform
9.1+ supports implementations running TCK
compatibility certification tests on JDK versions Java
SE 8, Java SE 11+. In support of running TCK tests on
JDK versions greater than SE 11, we expect that some
tests will need to be revised (e.g. see
jaxb-tck/issues/57 [1] for updating signature tests
related to need new signature tooling library and
signature map files).
Last December, we started
making changes to the `TCK Process 1.0` that includes
the following text [2] which introduces an alternative
to excluding (challenged) TCK tests:
`The specification project
may approve (user) workarounds for an `accepted` TCK
challenge (as alternative to excluding TCK tests).`
My question today is
whether the above quoted text is enough to cover
Jakarta EE 9.1 compatibility certification requests
against Java SE 17 (which will include signature test
failures due to jaxb-tck/issues/57 [1])? If the
answer/vote is yes, certain signature test failures
can be ignored on newer JDK versions, if and only if
the signature test failure is caused by inadequate TCK
signature support for the relevant Java SE (e.g. JDK
17) version. If the answer/vote is no, we will need
an additional TCK process change to specifically allow
a SPEC TCK service release that updates signature
tests to resolve the signature test failure (e.g.
allow jakarta-xml-binding-tck-3.0.2.zip [4] to be
officially released by Spec team so that
implementations can submit certification requests
against jakarta-xml-binding-tck-3.0.2.zip).
Scott
[1] https://github.com/eclipse-ee4j/jaxb-tck/issues/57
[2] https://github.com/jakartaee/jakarta.ee/pull/1018/files#diff-1fe254a18287c0db31fd9cb0a6bca11b1efda926095c3a65b73ef2ae0c89360dR223
[3] https://jakarta.ee/committees/specification/tckprocess/
[4] https://download.eclipse.org/ee4j/jakartaee-tck/jakartaee9-eftl/promoted/jakarta-xml-binding-tck-3.0.2.zip_______________________________________________
jakarta.ee-spec mailing list
jakarta.ee-spec@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec
_______________________________________________
jakarta.ee-spec mailing list
jakarta.ee-spec@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec
_______________________________________________
jakarta.ee-spec mailing list
jakarta.ee-spec@xxxxxxxxxxx
To unsubscribe from this list, visit https://urldefense.com/v3/__https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec__;!!ACWV5N9M2RV99hQ!awIHuBnS0vFrSdaLJF1CkeydaJ6HBDuZO-HU31C_-GlXemvLd-yxK7aypDQTQqQ$