Obviously we're not interested in either extreme. We're looking for
the right cost/benefit tradeoff.
If we think that users are going to be running the TCK against
vendor's products because they don't believe the product actually
passed the TCK, we have a much bigger problem.
The TCK is a very complex test suite (which doesn't use
Arquillian, by the way). Setting up and running the TCK against
someone else's product is a difficult task, even given all the
adapters that are needed. It's far more likely that people will do
it wrong and make incorrect claims that the product doesn't pass the
TCK.
The TCK adapter for a proprietary product may make use of
proprietary APIs that the vendor does not want exposed.
Or the proprietary product may come with a license that prohibits
independent testing of the product.
Or the product may only be usable on hardware or in environments
that most users won't have access to.
I can imagine many reasons a vendor might not want to make their TCK
adapter available to the public or available under an open source
license.
All that said, I'm sure many vendors will make their TCK
adapters available, and will make their TCK results available, so
that others can see exactly how they've tested their products. But
I think there's good reasons that some vendors may not want to do
that, and I don't see the benefit of requiring them to do it
to be great enough. The bar is already high enough to become a Java
EE / Jakarta EE vendor, we don't need to make it unnecessarily
higher.
Of course, if experience makes us suspect that vendors are cheating,
we can change the rules and require this. But in 20 years of Java
EE that hasn't been a problem.
James Roper wrote on 07/18/2018 08:39 PM:
If you push this to the extreme, you'll need to see the
source code for the vendor's product so you can make
sure they haven't implemented some special "TCK mode".
Obviously that's not feasible.
Well, if we're talking about extremes, why not go the
other way, and have no TCKs, and just trust each vendor at
their word that they've been careful and done their own
testing to ensure they implement the spec? But we're not
talking about extremes here, so there's no point in bringing
it up. Publishing an Arquillian adapter as open source is a
low cost, high value thing that can be done, extremes are
irrelevant when we're talking about such a cheap win. It
also provides a very simple starting point for the frank
discussions and compromises that you're saying should be
done when necessary, since it means everyone can start from
the same meaningful place, "we've run the TCK against
product X and found Y". Plus most, if not all Arquillian
adapters for Java EE containers out there are already open
source, so in most cases, adding such a requirement
primarily reinforces what is already done, it doesn't add
anything new.
In the end, cheating is dealt with in the courts. But
cheating is rare, and it rarely gets to that point.
Before then, there are plenty of opportunities for frank
discussion and compromises when necessary. Most vendors
are trying to do the right thing, and when the issues
are explained to them they'll do their best to follow
the rules. That's part of the reason that we have rules
that go with the TCK, not just tests, and when you sign
the license for the TCK you agree to follow the rules.
Mike
Milinkovich wrote on 07/18/2018 06:16 PM:
On
2018-07-18 9:05 PM, James Roper wrote:
A
discussion about verifying TCKs was cut short on the
Google docs, and that's probably because it's hard
to follow discussions in the tiny comments boxes, so
I wanted to start a thread here.
Firstly,
I'm not suggesting that Eclipse run the TCKs
themselves, that's obviously not feasible.
The
problem is that there is an assumption that just
because the TCKs are open, anyone can run them, and
so therefore anyone can verify whether a vendor is
falsifying test reports. This is not true. For
example, most MicroProfile TCKs are built on
Arquillian, which is a testing tool that abstracts
away an application server, allowing specs to
produce vendor agnostic TCKs. Arquillian itself is
open, however, the integration with a particular app
server is not part of Arquillian, and may or may not
be open. If the Arquillian integration for a
particular app server is not open and not freely
available, then only people that have access to the
Arquillian integration for that app server will be
able to run the TCK against it. This means, in such
a case, it won't be possible for anyone to verify
whether the TCK passed against that app server. Even
if the app server vendor provides the binaries for
Arquillian integration - we need more than that, we
need the source code of the Arquillian integration
to ensure that they haven't implemented any smoke
and mirrors in their integration. For example,
what's to stop them from providing Arquillian
integration that actually integrates to a different
vendors app server instead of theirs, in that case,
the TCK might pass, but only because it was testing
a different app server.
So
I think any implementation that wishes to claim
compatibility must, in addition to publishing test
results, also publish any integration code necessary
to run the TCK as open source. I would go as far as
to say that they should also provide scripts and
detailed instructions saying how to run the TCK,
since for some vendors app servers this might not be
at all trivial.
James,
Thanks for this. I now understand what you're trying
to say, and agree that it is a valid point.
I wonder if anyone from Oracle could chime in here
and comment on how this is handled currently with Java
EE certification?
![Avast
logo]()
|
This email has been checked for viruses by
Avast antivirus software.
www.avast.com
|
_______________________________________________
jakarta.ee-wg mailing list
jakarta.ee-wg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jakarta.ee-wg
_______________________________________________
jakarta.ee-wg mailing list
jakarta.ee-wg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jakarta.ee-wg
--
James Roper
Senior Developer, Office of
the CTO
_______________________________________________
jakarta.ee-wg mailing list
jakarta.ee-wg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jakarta.ee-wg
|
