Bill and I discussed this topic on the
jakartaee-tck PR for a while [1], but I believe it's time to try
and loop the discussion back to the mailing list to try and get
a final decision.
This has been discussed in the platform project team meeting
recently. The 2/18/2020 meeting minutes should appear here
shortly.
Thanks for bringing this up on the platform call. I was on vacation so I could not attend. From the meeting minutes I don't see any new information though -- just some questions which have been primarily discussed/answered in this thread already.
Overall it seems that the platform isn't going to force
specs to do TCKs a certain way, and that specs have the
freedom to choose.
I don't think that's the case.
We need a way to create the overall platform TCK, e.g., by combining
pieces from the standalone spec TCKs. To make that work, there's
going to need to be some agreement on how the pieces fit together.
For example, we'll likely need to agree on a testing framework, or
small set of frameworks, and some packaging rules, that allow the
integration.
Within the constraints of that framework(s), spec projects will be
free to develop their tests however they want. Hopefully some best
practices will emerge and we'll learn from each other and come out
with something even better.
Among JSON-B committers we are in agreement that we want to
migrate to do TCKs this way, and JSON-P would also like to
follow suit [2].
I'm happy to use JSON-B as a guinea pig to see how it works
and react to feedback.
At this point I don't see any remaining blockers for moving
forward with the initial concept, but I will wait for a few
days to let people chime in.
I think it's fine to run some experiments such as this. They may
help us learn some of the things we'll need to know to do the much
bigger project. We don't want to end up with two TCKs that you need
to run for each spec, so all of this needs to come back together at
some point.
Note however that these two cases are some of the simpler cases.
They're effectively just standalone libraries with minimal
dependencies and minimal run time requirements. When we get to
things like Jakarta Mail, which depends on an external mail server,
and JAX-RS, which is a client/server API, the framework and
integration requirements will be more important.
Now we seem to be going in circles... To recap the relevant points:
1) How do we create overall platform TCK and how does it fit together?
> In the same way that CDI and BeanValidation did it for JavaEE 8 / JakartaEE 8
2) What test framework? What technologies?
> Arquillian+JUnit. These technologies are well-tested and widely understood in the community, and are perfectly capable of doing more complex testing such as pulling in DBs, mail servers, or other external resources. CDI and BeanVal are reasonably complex specs/TCKs and are already using Arquillian.
3) [From the meeting minutes] Requirement – allow the use of the existing TCK tests themselves. If all of the TCK tests need to be modified just to become part of this new infrastructure, the process will die. We need the ability to incorporate existing TCK tests.
> The conversion can be done on a spec-by-spec case. For example, everything would work fine if just JSON-B migrated to the "external TCK" way and all other specs remained as-is. Also, the conversion could be all-in-one or partial. For JSON-B I did an all-in-one conversion, meaning I took _all_ existing TCK tests and converted them to Arquillian+JUnit so now JSON-B TCK tests don't use JavaTest at all. Personally I think per-spec all-in-one conversion is best so there is only ever 1 framework per spec, but each spec could have the flexibility to do what they want.
Andy
Guibert wrote on 2/6/20 5:54 PM:
> Right now we have 2 types of TCKs:
> A) CDI and BeanVal use Arquillian+TestNG, and produce
their TCKs as maven
> dependencies
> B) All other specs use JavaTest
>
> I'm proposing to make a 3rd type of TCK (very similar to
A) that uses
> Arquillian+JUnit.
Which would only be used by completely new TCKs? You don't
plan to convert any
existing TCKs to this style?
> However, I don't really care what test framework gets
used to run the TCKs,
> the main items I care about as a spec developer are:
> 1) API/spec/TCK all in one repo
> 2) Implementations (standalone lib or app server) can
_easily_ pull in and run
> the TCKs as a Maven Dependency and doing `mvn test` or
`mvn verify`
This looks like another tabs vs. spaces issue. :-)
All I care about is that the API, spec, and TCK are each in
their own repo. But
we decided not to force either style on projects; projects can
structure this in
repos however they feel is best.
Most of our tests aren't produced as Maven artifacts and
aren't runnable from
Maven so that's going to take a lot of work. One of the
challenges is that the
"official" signed versions of the TCKs are produced as
standalone zip files.
Also, configuring some of these TCKs is significantly
non-trivial. It's now
clear that you'll ever be able to run them with just "mvn
test".
> This is what MicroProfile has done and the model works
very well.
>
> Also, I'm not suggesting we mandate _all_ specs migrate
to this way (or
> require all specs to do anything). Rather, I'm suggesting
that we tolerate
> specs to migrate to the way CDI/BeanVal does it (except
using JUnit instead of
> TestNG preferably).
I don't believe there's any restriction preventing new specs
from doing so.
