Hi Jan, I think you've raised some good questions -- I've picked out 2
basic questions from your note which I will try and answer.
1) How/where do we test multi-spec interactions?
Lets say we have an interaction between CDI + JSONB that needs to be
tested. I propose we pick one of the specs of the involved
technologies to house the test. This way we don't have any TCK tests
in "no mans land". If there are optional integrations (e.g. do X IF
the runtime is CDI-capable) we can control this in a number of ways
such as: A) do a Class.forName() of some CDI API to see if CDI is
available in the test or B) control test inclusion with sysprops/env
vars like `mvn test -Dcdi=false` to disable CDI-capable tests.
2) What standard test framework do we settle on? We don't want every
spec doing their own thing because it will make running all TCK tests
very difficult.
Fully agree. I propose we do the following:
- Create a new entry point in the jarkartaee-tck repo which will be a
maven-based project that can test the new "externalized" TCK tests.
Then we can incrementally migrate TCK tests on a per-spec level to the
new externalized form.
- Externalized TCK tests MUST use JUnit + Arquillian, even if the
technology is standalone (like JSONB). This way, application servers
can still certify that the "standalone" technologies still work when
deployed in applications on the servers. We can create a "no-op"
Arquillian container that standalone implementations can use (e.g.
Yasson for JSONB) for when components want to verify TCK tests in
standalone mode. Although MicroProfile does not have a deployment
spec, its TCK tests still use Arquillian to create JEE deployments
such as .war or .ear files.
- Create a new umbrella entry point so we can run all TCK tests in
one shot: the old ones based on the JavaTest harness, and the
new/externalized ones based on JUnit+Arquillian
- Andy
On Mon, Feb 3, 2020 at 11:53 AM Jan Supol <jan.supol@xxxxxxxxxx
<mailto:jan.supol@xxxxxxxxxx>> wrote:
> +1 Kevin! I think the TCK in Jakarta raises the bar higher for
people
> to run TCK freely. As a contributor, I would like to be able to add
> TCKs whenever I propose API/Spec changes. Otherwise, we will run
out
> of sync in no time. Therefore, I support what Andy proposed to
split
> TCK and rehome them to be with where it should belong. I speak
> directly from my own experience of working on MicroProfile. It is
> great to add tests together with spec or api changes to demonstrate
> the full piece.
This is exactly what I miss on Microprofile, the tests that are
mandatory if and only if two of the specs should work together
(i.e. not
optional tests). Those were known as the CTS only tests, i.e. were
not
part of the standalone TCKs (For instance CDI & JAX-RS, CDI &
WebSocket,
BeanValidation & JAX-RS). Standalone TCKs are what the Microprofile
has, it does not have the full platform test. If the tests should
still
be part of the platform TCK, and the tests are in the Spec
repository,
how to make sure the tests are run for full platform and not for the
standalone TCK?
Using the approach Andy used for JSON-B works fine. What the approach
does not have, and what Microprofile does not have, is the
deployment.
Jakarta EE TCK has the concept of deployment (not necessarily
JSR-88),
and the deployment behaviour is described in various Specs, such as
Servlet of WebSockets. How the concept of the deployment would
work with
Andy's approach is not clear. Should the deployment be tested in the
future? Should it be left on the Spec to define the deployment
requirements/API or should it more common to every Spec?
Currently, the
TCK framework does that and vendors implement the porting package
that
takes care of the deployment. Every TCK uses the same mechanism. But
with having a different test framework in each TCK, the deployment
needs
to be thought through.
As for releasing TCKs separately:
> 1. When releasing a new version of a spec, we release
API/Spec/TCK and
> they have either major or minor version changes. They share the
same
> version. With the same version, we know which version of TCK we
should
> execute for the version of Spec you choose.
> 2. If we found any bugs in TCK, we can do a patch release for the
> project, which produces a micro/patch version release of
API/Spec/TCK
> so that runtimes can execute the new TCK while end users are not
> affected.
> The above fits in well with the general understanding of
> major/minor/micro release.
It is great that you think about standalone TCK. However, that is the
easy part. The hard part is to think about the full platform. The
approach Andy described is to JUnitize the tests. The other approach
will use some different framework. It is fairly fine to have separate
TCKs like this as long as the Full platform TCK is no longer a
requirement. Before the split, it needs to be known whether the full
platform TCK is what is required, or whether the set of standalone
TCKs,
such as in Microprofile, is enough. If the full platform TCK is
needed
for verifying the compatible implementation, the separation needs
to be
done with the idea of putting all parts together. Otherwise, each
Spec
will take their part, convert it to use some fancy test framework and
say: "Hey, its not my business to make the full platform TCK
possible.
Someone else should take care".
Even the Microprofile has a unified Arquillian based test
environment.
Using JUnit in one TCK, Arquillian in other, and JavaTest in the next
brings a challenge for the team that ought to run the TCKs. Are we
cool
with having it different in every TCK?
> On Mon, Feb 3, 2020 at 1:40 PM Kevin Sutter <sutter@xxxxxxxxxx
<mailto:sutter@xxxxxxxxxx>
> <mailto:sutter@xxxxxxxxxx <mailto:sutter@xxxxxxxxxx>>> wrote:
>
> Jan,
> You bring up some excellent points. Having a well thought out
> plan before jumping in with both feet would be nice. But, on the
> other hand, we don't want to stifle the excited effort that Andy
> is demonstrating. It's a tough line. Maybe we need to allow one
> or two teams to experiment with the processes and see what it
> takes. And, then we can take those experiences and develop
a plan
> that works across all the projects.
>
For the experiment, grabbing JSON-B, or JSON-P is easy. Those are
basically standalone only technologies. A Servlet based technology
seems
more appropriate for experimenting.
>
>
> ---------------------------------------------------
> Kevin Sutter
> STSM, MicroProfile and Jakarta EE architect @ IBM
> e-mail: sutter@xxxxxxxxxx <mailto:sutter@xxxxxxxxxx>
<mailto:sutter@xxxxxxxxxx <mailto:sutter@xxxxxxxxxx>> Twitter:
> @kwsutter
> phone: tl-553-3620 (office), 507-253-3620 (office)
> LinkedIn: https://www.linkedin.com/in/kevinwsutter
>
>
>
> From: Jan Supol <jan.supol@xxxxxxxxxx
<mailto:jan.supol@xxxxxxxxxx> <mailto:jan.supol@xxxxxxxxxx
<mailto:jan.supol@xxxxxxxxxx>>>
> To: jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>
> <mailto:jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>>
> Date: 01/31/2020 15:02
> Subject: [EXTERNAL] Re: [jakartaee-platform-dev] TCK tests
in the
> same repo as API and Spec
> Sent by: jakartaee-platform-dev-bounces@xxxxxxxxxxx
<mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx>
> <mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx
<mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx>>
>
------------------------------------------------------------------------
>
>
>
> The discussion on the various email lists is quite
extensive, and I
> admit I do not follow every thread that is discussed. But I am
> waiting
> for the plan from any of the committee we have since the
> transition to
> Eclipse Foundation. And I am not aware about any clear statement
> about
> what the final goal is. For start:
>
> - Is the splitting really what is recommended?
>
> - Should the Platform Jakarta EE TCK still be available in
the future
> after the split?
>
> - Should the interoperability between Specs tests (the tests
that
> require multiple Specifications to cooperate as it is
described in
> one
> or the other specification, the tests that used to be part
of the
> full
> CTS only) still be part of the combined (?) Jakarta EE TCK?
>
> - Is there a recommended framework to be used in the future
with the
> TCKs (such as the existing JavaTest + perhaps Arquillian
similarly to
> Microprofile + ?)?
>
> Those should be goals presented by the Jakarta
representatives before
> the TCK repo is started to be ripped apart. Not on email, but
> clearly on
> the main Jakarta informational page (whatever that is now).
>
> -- Jan
>
> On 31.01.2020 20:34, Lance Andersen wrote:
> > I also agree with Bill. There has to be thought given to CTS
> > structure first IMHO so that there is consistency in the test
> > structure to make it easier to pull in the standalone
tests and
> > where the platform specific tests are maintained for the
> standalone
> > technologies.
> >
> > This will take some time to flush this out. During the Java EE
> days,
> > we did find it easier to work out of one master workspace, but
> part of
> > that was due to the fact there was one team primarily
> responsible for
> > the test development for all of the Java EE technologies.
> >
> >> On Jan 31, 2020, at 2:01 PM, dmitry.kornilov@xxxxxxxxxx
<mailto:dmitry.kornilov@xxxxxxxxxx>
> <mailto:dmitry.kornilov@xxxxxxxxxx
<mailto:dmitry.kornilov@xxxxxxxxxx>>
> >> <mailto:dmitry.kornilov@xxxxxxxxxx
<mailto:dmitry.kornilov@xxxxxxxxxx>> wrote:
> >>
> >> Bill, you are right. On the other hand the process of
splitting
> has
> >> to be started somehow. It will never start with an
assumption that
> >> some developer with TCK knowledge will take an initiative and
> start
> >> working on it. There are not too many developers like
this and
> no one
> >> wants to take a risk and responsibility.
> >> On the other hand I don’t think that it’s a right time to
do it
> now
> >> as part of Jakarta EE 9 release. I already proposed
thatwe will
> not
> >> move JSONB TCK tests from CTS yet. We will do it after
Jakarta
> EE 9
> >> is released. We will keep them in sync until that time. Users
> may use
> >> TCK tests in JSONB repo as more convenient way of checking
> >> compatibility. But it cannot be used officially for
compliance
> >> testing for now.
> >> -- Dmitry
> >> *From:*jakartaee-platform-dev-bounces@xxxxxxxxxxx
<mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx>
> <mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx
<mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx>>
> >>
> <mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx
<mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx>><jakartaee-platform-dev-bounces@xxxxxxxxxxx
<mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx>
> <mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx
<mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx>>
> >> <mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx
<mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx>>>*On Behalf
> >> Of*Bill Shannon
> >> *Sent:*Friday, January 31, 2020 7:06 PM
> >> *To:*jakartaee-platform developer discussions
> >> <jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>
> <mailto:jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>>
> >> <mailto:jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>>>; Andy Guibert
> >> <andy.guibert@xxxxxxxxx <mailto:andy.guibert@xxxxxxxxx>
<mailto:andy.guibert@xxxxxxxxx <mailto:andy.guibert@xxxxxxxxx>>
> <mailto:andy.guibert@xxxxxxxxx <mailto:andy.guibert@xxxxxxxxx>>>
> >> *Subject:*Re: [jakartaee-platform-dev] TCK tests in the same
> repo as
> >> API and Spec
> >>
> >> This has been discussed numerous times over the last 3 years.
> No one
> >> disagrees with the goal of splitting up the TCK repo.
Whether the
> >> individual spec TCKs are in the same repo as the API
classes or
> in a
> >> different repo is just a detail. The challenge is in
splitting up
> >> the existing TCK repo such that the resulting TCK for
each spec is
> >> functionally identical to the existing standalone TCKs
for the
> specs,
> >> and that the platform TCK is somehow created by combining
all the
> >> individual TCKs to produce a new platform TCK that is
functionally
> >> identical to the existing platform TCK.
> >>
> >> Needless to say, this is not a small job.
> >>
> >> No one with deep knowledge of the existing TCKs has come
> forward with
> >> a detailed plan for how to achieve the above. Without
that, we're
> >> all just wishing and hoping.
> >>
> >> And I strongly encourage you to*not*just start hacking on the
> TCK to
> >> create what you want for your spec. We need to solve the
larger
> >> problem and a bunch of uncoordinated hacks to individual
TCKs will
> >> not get us there.
> >>
> >> Thanks.
> >>
> >> P.S. You should be able to find some of the previous history
> of this
> >> discussion in the jakartaee-tck-dev mailing list.
> >>
> >> Andy Guibert wrote on 1/30/20 10:45 AM:
> >>> Hi all,
> >>> Currently all of the Jakarta EE TCK tests are housed in one
> big repo
> >>> and they use a custom test framework from the Java EE days:
> >>> https://github.com/eclipse-ee4j/jakartaee-tck
> >>> It is more convenient to have the TCK tests in the same
repo
> as the
> >>> API and spec docs because as the technologies change
over time
> all 3
> >>> parts (tck/api/spec) can be updated in the same PR.
Additionally,
> >>> implementations can then consume the TCK tests as Maven
artifacts
> >>> and run/verify that they pass the TCK. This is what
> MicroProfile has
> >>> done and it works very well.
> >>> As an example, I've started this off with the JSON-B TCK
test
> here:
> >>> jsonb-api pr:
https://github.com/eclipse-ee4j/jsonb-api/pull/221
> >>> yasson (impl) pr:
https://github.com/eclipse-ee4j/yasson/pull/379
> >>> Just wanted to share this with the wider dev community and
> encourage
> >>> other specs to follow suit as time permits.
> >>> Cheers, Andy
> >>> -- IBM
> >>>
> >>>
> >>> _______________________________________________
> >>> jakartaee-platform-dev mailing list
> >>> jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>
> <mailto:jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>>
> <mailto:jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>>
> >>> To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> >>>
https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
> >> _______________________________________________
> >> jakartaee-platform-dev mailing list
> >> jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>
> <mailto:jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>>
> >> <mailto:jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>>
> >> To change your delivery options, retrieve your password, or
> >> unsubscribe from this list, visit
> >>
https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
> >
> > <http://oracle.com/us/design/oracle-email-sig-198324.gif>
> >
>
<http://oracle.com/us/design/oracle-email-sig-198324.gif><http://oracle.com/us/design/oracle-email-sig-198324.gif>
> > <http://oracle.com/us/design/oracle-email-sig-198324.gif>Lance
> > Andersen| Principal Member of Technical Staff |
+1.781.442.2037
> > Oracle Java Engineering
> > 1 Network Drive
> > Burlington, MA 01803
> > Lance.Andersen@xxxxxxxxxx
<mailto:Lance.Andersen@xxxxxxxxxx>
<mailto:Lance.Andersen@xxxxxxxxxx <mailto:Lance.Andersen@xxxxxxxxxx>>
> <mailto:Lance.Andersen@xxxxxxxxxx
<mailto:Lance.Andersen@xxxxxxxxxx>>
> >
> >
> >
> >
> > _______________________________________________
> > jakartaee-platform-dev mailing list
> > jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>
> <mailto:jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>>
> > To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> >
https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
> _______________________________________________
> jakartaee-platform-dev mailing list
> jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>
> <mailto:jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>>
> To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
>
>
>
>
> _______________________________________________
> jakartaee-platform-dev mailing list
> jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>
> <mailto:jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>>
> To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
>
>
>
> --
> Thanks
> Emily
>
>
> _______________________________________________
> jakartaee-platform-dev mailing list
> jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>
> To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
<mailto:jakartaee-platform-dev@xxxxxxxxxxx>
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
