[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
Re: [jakartaee-platform-dev] [EXTERNAL] Re: TCK tests in the same repo as API andSpec
|
Some of the proposed solutions depend on an adapter. I don't know
that we want to try for a universal adapter, or even a large number
of adapters. If the adapter approach is feasible, I would guess
that we would still want to limit it to a very small number of test
suite styles for developing the individual tests.
I don't want to boil the ocean, I just want to cook dinner.
Scott Kurz wrote on 2/24/20 12:15 PM:
I haven't been able to get a handle on the overall thread.
But let me just start with the Batch TCK, which is already using
a TestNG suite.
If there were a way to "adapt" this suite so that the CTS / JT
harness driving it could use whatever
- executing with filtering (run these tests, not those)
- reporting/logging
then it'd seem like we'd achieve a lot of the goals: allowing TCK
developers to develop with their preferred test library/framework,
not requiring rewrites.
Did I miss anything? Any ideas on where to start with such an
adapter?
------------------------------------------------------
Scott Kurz
WebSphere Batch and Developer Experience
skurz@xxxxxxxxxx
--------------------------------------------------------
"Kevin Sutter" ---02/24/2020 02:57:39 PM---I
have tried to post this to the jakartaee-tck-dev mailing list
since that was the action from last
From: "Kevin
Sutter" <sutter@xxxxxxxxxx>
To: jakartaee-platform
developer discussions <jakartaee-platform-dev@xxxxxxxxxxx>
Date: 02/24/2020
02:57 PM
Subject: Re:
[jakartaee-platform-dev] [EXTERNAL] Re: TCK tests in the same
repo as API andSpec
Sent by: jakartaee-platform-dev-bounces@xxxxxxxxxxx
I have tried to post this to the jakartaee-tck-dev
mailing list since that was the action from last week's Platform
call. And, since Andy is referencing some of it below, I thought
I would start off my reply with a pointer to the minutes: https://eclipse-ee4j.github.io/jakartaee-platform/minutes/2020-02-18.html. And, specifically the section of the minutes where we
discussed the TCK effort.
Criteria for replacing the TCKs. [BS]
- Discussed how to ensure that a “refactored” TCK is a
sufficient replacement to the original (previous version)
- Is each individual Spec project responsible to verify the
re-factoring? Is that sufficient? Or, do we need some
external checks-and-balances?
- In the past (J2EE, Java EE), the TCKs were incrementally
modified. Easier to monitor the changes going in.
- Structured reviews might help ensure consistency.
Participants from Spec Project team, the Platform TCK team,
the Spec Project TCK team are required.
- Common framework for the TCKs? Or, allow each independent
TCK to determine the framework used? A common framework is
probably key to the success of this effort.
- Defining a common framework would allow each Project to
plug in their TCK and be executed as part of the overall
Platform TCK.
- Requirement – allow the use of the existing TCK
tests themselves. If all of the TCK tests need to be
modified just to become part of this new infrastructure, the
process will die. We need the ability to incorporate
existing TCK tests. Maybe that’s through some build magic or
wrappering of the tests or something…
- Excellent start to this discussion, but needs much more
work. Who should drive this effort? Platform TCK? An
individual TCK (ie. json-b or json-p)? Group effort? Post to
the Platform TCK mailing list and ask for volunteers. (KWS)
Although the discussion on the call was very good, it also
demonstrates that we don't have a complete solution yet. The
thought on the call is that we should try to enlist some people
with TCK experience to help with defining the solution. Since
the discussion seems to want to live in the Platform mailing
list instead of the TCK mailing list, I'll post here instead...
I think the main Requirement (highlighted above) that is getting
missed in the discussion is the need to have a framework defined
that can easily incorporate existing testcode into a cohesive
test suite.
As stated above, if we define a process that requires updates to
existing test code source, then the overall goal of separating
the TCKs will die. We'll get to the 10-12 gung-ho projects that
do the separation, but if it's not an easy process to move
forward then the other projects won't find the cycles to do the
work. And, we definitely do not want to have duplicate test
source in multiple repos. It's fine for now for json-b and
json-p as we continue to experiment, but we don't want that
practice to continue. It's great to have the individual tests
run with each project, but we also need an easy way to
incorporate those same tests into an overall TCK test suite.
Andy, I'm not sure if you are signing up for this? Or, maybe you
can find some existing TCK expertise to help you out? But, the
end goal is to make this a well-defined process for
incorporating separate TCKs into the overall Platform TCK.
---------------------------------------------------
Kevin Sutter
STSM, MicroProfile and Jakarta EE architect @ IBM
e-mail: sutter@xxxxxxxxxx Twitter: @kwsutter
phone: tl-553-3620 (office), 507-253-3620 (office)
LinkedIn: https://www.linkedin.com/in/kevinwsutter
From: Andy Guibert
<andy.guibert@xxxxxxxxx>
To: Bill Shannon
<bill.shannon@xxxxxxxxxx>
Cc: jakartaee-platform developer
discussions <jakartaee-platform-dev@xxxxxxxxxxx>
Date: 02/24/2020 12:38
Subject: [EXTERNAL] Re:
[jakartaee-platform-dev] TCK tests in the same repo as API
andSpec
Sent by: jakartaee-platform-dev-bounces@xxxxxxxxxxx
On Sun, Feb 23, 2020 at 12:16 PM Bill Shannon <bill.shannon@xxxxxxxxxx>
wrote:
Andy Guibert wrote on 2/20/20 11:52 AM:
Now we seem to be going in circles... To recap the relevant
points:
1) How do we create overall platform TCK and how does it fit
together?
> In the same way that CDI and BeanValidation did it for JavaEE
8 / JakartaEE 8
They didn't. That's the point. That's what we want to avoid.
It's true that CDI/BV do not roll up into the same spot as
everywhere else, but based on my proposed PR
to jakartaee-tckI thought we didn't want that
for externalized TCKs?
Speaking from experience here, in IBM/OpenLiberty we have an
entire team dedicated to setting up and running the JEE CTS. They
are the only ones who know how to set up and run the arcane ball
of JakartaEE TCKs.
When I was working on the BeanValidation 2.0 implementation for
OpenLiberty for JavaEE 8 certification I was able to figure out
how to set up and run the BeanValidation 2.0 TCKs against
OpenLiberty in under an hour because I was already familiar with
Arquillian (as many JEE developers already are). On the other hand
only the dedicated CTS teams know how to set up and run the
JavaTest stuff.
2) What test framework? What technologies?
> Arquillian+JUnit. These technologies are well-tested and
widely understood in the community, and are perfectly capable of
doing more complex testing such as pulling in DBs, mail servers,
or other external resources. CDI and BeanVal are reasonably
complex specs/TCKs and are already using Arquillian.
How do we teach Arquillian to run the existing test cases? Or
how do we convert the existing test cases so that Arquillian can
run them? And either way, how do we assure ourselves that the
result has the same pass/fail characteristics as the original?
I converted the existing JSON-B test cases to Arquillian in this
commit here: https://github.com/eclipse-ee4j/jsonb-api/pull/221/commits/6a4867bd18a6b88750066ac43debbedfce1bebb4
The changes were pretty systematic and I did some spot negative
testing to ensure validity.
Ultimately, we assure ourselves the results are the same by
relying on our developers and our code reviewers. This is a
one-time change per-spec and coming up with a more elaborate
process would be over-engineering a solution IMO.
3) [From the meeting minutes] Requirement – allow the use of the
existing TCK tests themselves. If all of the TCK tests need to
be modified just to become part of this new infrastructure, the
process will die. We need the ability to incorporate existing
TCK tests.
> The conversion can be done on a spec-by-spec case. For
example, everything would work fine if just JSON-B migrated to
the "external TCK" way and all other specs remained as-is. Also,
the conversion could be all-in-one or partial. For JSON-B I did
an all-in-one conversion, meaning I took _all_ existing TCK
tests and converted them to Arquillian+JUnit so now JSON-B TCK
tests don't use JavaTest at all. Personally I think per-spec
all-in-one conversion is best so there is only ever 1 framework
per spec, but each spec could have the flexibility to do what
they want.
This is the process that the notes claim will die. Depending on
each spec project to convert all of their tests to a new framework
just won't happen. You might end up with 10 specs fully converted
and 27 specs still using the old framework. If old and new can
coexist, that might be fine. Even if we had to run two completely
separate TCK frameworks, that might be acceptable. I'd like to
hear what others think about this entire process of converting
tests to a new framework, whether it needs to be done
automatically, whether it can be done manually and incrementally,
etc.
Yes, 10 specs could convert to the new way and 27 specs could stay
with the old framework and that would be perfectly fine. IMO
That's a win-win because the 10 specs that took the time to
convert get the quality-of-life boost that comes with the
decoupled/modernized and the 27 specs that stayed with the old way
continue on with business as usual.
_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://urldefense.com/v3/__https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev__;!!GqivPVa7Brio!PMMpIrGKQhzDDh1xagIMc2U8M8GLyiW2-vmNwhxTAX38yOfoV9e2RlE-qXTlZqRTow$