Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jakartaee-platform-dev] [EXTERNAL] Re: TCK tests in the same repo as API andSpec
Bill, I had a sit down meeting with the CTS lead at IBM and we discussed the pros and cons of the current way and the "externalized" way that I have proposed. Ultimately, he said that we already have both ways working and automated and it didn't make a big difference which way each spec goes.

I see two types of stakeholders with the TCK tests:
 1) People who work on moving specs forward and writing new TCK tests, including getting tests passing for new versions of specs
 2) People who run/maintain the TCK tests long term to ensure things don't regress

Based on the opinions I've gathered from people in role (2) in this thread and from direct discussions at IBM, they don't care one way or another.
Based on the opinions I've gathered from people in role (1), they prefer the "externalized" way of testing. Furthermore, this decision of what type of testing to go with can be made at a per-spec level so it doesn't need to be a unanimous platform decision. For people in role (1) for JSON-B, it is unanimous that we want to migrate to the "externalized" TCK way, and we have already put in the work to do so.

If anyone from other companies have concrete concerns with this approach (Oracle or others) I think it would be best to set up a Webex with the concerned parties so we can talk it over and report back to this thread.

On Thu, Feb 27, 2020 at 6:32 PM Bill Shannon <bill.shannon@xxxxxxxxxx> wrote:
It looks like this is testing a requirement from the JAX-RS spec,
so ideally it would be included in the JAX-RS TCK and would be enabled
only when running against a product that includes both JAX-RS and EJB.

For now, that might only be a platform product, so leaving it in the
platform TCK would probably be an acceptable compromise.

Scott Marlow wrote on 2/26/20 11:45 AM:
> On Wed, Feb 26, 2020 at 11:57 AM Bill Shannon <bill.shannon@xxxxxxxxxx> wrote:
>>
>> Some specs will have conditional requirements ("do this when running with CDI") and should own the tests for those requirements.
>>
>> Still, there are some requirements that are only in the platform spec (e.g., deployment requirements) and those tests should be owned by the platform spec project.
>>
>> But I agree that every test needs to be owned by some spec project.
>
> Should https://urldefense.com/v3/__https://github.com/eclipse-ee4j/jakartaee-tck/tree/master/src/com/sun/ts/tests/jaxrs/platform/ejbstateless__;!!GqivPVa7Brio!IaQzE2J_YxOGWF8ryf10HQXL5eSuC4OztpC2U8UisxRgZ87DDaHxDooxRwuwPwSYcg$
> stay in the platform TCK since they depend on a platform
> implementation?
>
>>
>> Andy Guibert wrote on 2/26/20 7:44 AM:
>>
>> Regarding "platform tests" I'd like to move away from tests that are "owned" by the platform and instead have multi-spec integrations be "owned" by one of the specs involved.
>>
>> For example, if we have a test for a BeanValidation + CDI interaction, the let the test be housed in the BeanValidation TCK.  By default, when you run the BeanValidation TCK _all_ tests would run (including the BV+CDI ones), and if an implementation did not want to certify with CDI then they could explicitly disable the CDI tests by doing something like `mvn test -Dcdi=false`.
>>
>> I think it will be helpful to not have tests in "no mans land" for the sake of test development and maintenance.
>>
>> On Wed, Feb 26, 2020 at 8:34 AM Scott Marlow <smarlow@xxxxxxxxxx> wrote:
>>>
>>> On Tue, Feb 25, 2020 at 6:52 PM Andy Guibert <andy.guibert@xxxxxxxxx> wrote:
>>>>
>>>>
>>>>
>>>> On Tue, Feb 25, 2020 at 11:35 AM Bill Shannon <bill.shannon@xxxxxxxxxx> wrote:
>>>>>
>>>>> In my opinion, shell script integration "doesn't count".
>>>>>
>>>>> I know how to write a shell script that runs A and then runs B.  That's the easy part.
>>>>>
>>>>> The integration I'd like, and that we may not be able to achieve, is to only have to configure the TCK once - where is the app server, how do I deploy to it, where is the database server, where is the LDAP server, where will I find log file output, how do I configure which tests to run, etc.
>>>>
>>>>
>>>> After speaking with some of the CTS experts in IBM, I would contest the claim that we currently "only have to configure the TCK once".
>>>> All of the various configs for the TCK are in one giant (over 2.5k lines) config file:
>>>> https://urldefense.com/v3/__https://github.com/eclipse-ee4j/jakartaee-tck/blob/master/install/jakartaee/bin/ts.jte__;!!GqivPVa7Brio!IaQzE2J_YxOGWF8ryf10HQXL5eSuC4OztpC2U8UisxRgZ87DDaHxDooxRwvpY2O_4A$
>>>> Additionally, each spec can also define its own config extensions like so:
>>>> https://urldefense.com/v3/__https://github.com/eclipse-ee4j/jakartaee-tck/blob/master/install/jaxrs/bin/ts.jte__;!!GqivPVa7Brio!IaQzE2J_YxOGWF8ryf10HQXL5eSuC4OztpC2U8UisxRgZ87DDaHxDooxRwvx_r2thg$
>>>>
>>>> I actually see it is a drawback, rather than a benefit, that config for the entire platform TCK is in one place. For example, if I just want to run the TCKs for one spec on my company's runtime, I'd have to sift through the entire 2.5k line "common" config file and configure the parts that my spec's TCK needs.
>>>>
>>>
>>> If we include the platform level TCK tests in that one spec, then that
>>> spec TCK will need platform level configuration, so the TCK can run
>>> all of the platform level tests for that technology.  Whether it is
>>> one big configuration file that includes everything, or a smaller one
>>> that includes enough configuration for running the EE server
>>> implementation and that spec, IMO the file will likely be large anyway
>>> because of the server settings.
>>>
>>> Scott
>>>
>>> _______________________________________________
>>> jakartaee-platform-dev mailing list
>>> jakartaee-platform-dev@xxxxxxxxxxx
>>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>>> https://urldefense.com/v3/__https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev__;!!GqivPVa7Brio!IaQzE2J_YxOGWF8ryf10HQXL5eSuC4OztpC2U8UisxRgZ87DDaHxDooxRwtwPGiUrg$
>>
>>
>> _______________________________________________
>> jakartaee-platform-dev mailing list
>> jakartaee-platform-dev@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://urldefense.com/v3/__https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev__;!!GqivPVa7Brio!LHqluJ3CA_SJnMgf4yY_EqJ_1eN0uH3BU59L4tSTO0yT8LcBffOfzvrPs4MQc7WjiQ$
>>
>>
>> _______________________________________________
>> jakartaee-platform-dev mailing list
>> jakartaee-platform-dev@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://urldefense.com/v3/__https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev__;!!GqivPVa7Brio!IaQzE2J_YxOGWF8ryf10HQXL5eSuC4OztpC2U8UisxRgZ87DDaHxDooxRwtwPGiUrg$
>
> _______________________________________________
> jakartaee-platform-dev mailing list
> jakartaee-platform-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://urldefense.com/v3/__https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev__;!!GqivPVa7Brio!IaQzE2J_YxOGWF8ryf10HQXL5eSuC4OztpC2U8UisxRgZ87DDaHxDooxRwtwPGiUrg$
>
_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev

Back to the top