Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jakartabatch-dev] TCK directions / form factor , e.g. Arquillian - opinions wanted
The idea of using JUnit 5 as a replacement of Arquillian is tempting. I personally would like that Arquillian itself supported JUnit 5 natively with a JUnit extension so that it can just be simply added to any JUnit 5 test to run it in a container. But we don't need to introduce JUnit 5 in a single step.

Over the weekend I had a deeper look at both the standalone TCK and the Batch part in the Jakarta CTS. 
After thinking about it for a while, I suggest the following, for the reasons explained below:

1. convert the existing Test NG tests into Arquillian tests in such a way, that we abstract the Arquillian-specific code into abstract test classes that all other tests would extend. This should be simple to do and would require negligible refactoring. We can create an example project to run SE implementations that will run JBatch using the Weld Arquillian container. After this, we could discard the tests in the Jakarta CTS and use the same test suite for both SE and EE.
2. explore how we could refactor to using JUnit 5. If we find a good solution, it would then be possible to achieve all the advantages of Romain's suggestion and also provide an example for other projects how they can modernize their TCKs with JUnit 5. It could also provide a good feedback for the Arquillian JUnit 5 integration that could help improve the Arquillian project.

The tests are written in Test NG and a conversion to Arquillian + Test NG would solve how to run them in EE, outside of the Jakarta CTS. Exactly what Bean Validation and CDI TCKs do now. However, running non-EE implementations would be more complicated as now and require configuring either a Java SE or Weld container. There's also a problem of packaging, as Arquillian requires a deployment package, which is a WAR or EAR in EE but usually plain JAR in SE. This is also an issue we faced in MicroProfile TCKs where most of the TCKs create a WAR package and any SE implementation needs to create an Arquillian extension that repackages it to JAR before running the tests.

On the other hand, rewriting the tests to JUnit 5 as Romain suggests, allows more flexibility and make it easier to develop new tests. The Batch TCK doesn't need much from from Arquillian, only packaging (which is a bit flawed as I explained), deployment and running in CDI environment. This can be supported in a different way. However, as I assumed, Romain's example assumes a lot of work on the integration side, would require the InvocationInterceptor feature available since Junit 5.5 and most EE integrations would still most probably rely on their Arquillian connectors rather than writing an JUnit 5 extension from scratch (I really didn't see any direct JUnit 5 support for any app server yet despite that Romain says that "all servers are getting junit 5 integrations" ). We could mitigate it if we provided e.g. some helper classes in the TCK to simplify the integration.

For those who didn't get Romain's idea with JUnit 5, it's basically using JUnit 5 with its extension points instead of Arquillian. Integrators would write a JUnit 5 extension that would do what server connectors do in Arquillian. So they could be just a thin wrapper over an existing Arquillian connector but SE implementations would just need a very simple extension to configure the impl or even no extension at all. But a JUnit 5 extension only gets general meta information about the test. The TCK would probably need to provide some information via JUnit tags or helper methods to inform which classes and resources to pack.

Ondro


pi 20. 3. 2020 o 9:00 Romain Manni-Bucau <rmannibucau@xxxxxxxxx> napísal(a):
Hi everyone,

Before giving some code pointers i'd like to mention I agree TCK are very low priority IMHO (users don't like that part anyway in general because it does not bring any feature and it should be global for Jakarta probably).

Now to answer Scott's question, I didn't do a full impl but just a structure of (java) code with some technical points to illustrate what I have in mind: https://github.com/rmannibucau/batch-tck/commit/18088a4a2445c6a72b2e62f9f1ea583f6057bddb
Here are the highlights:

1. Container must impl a JUnit 5 extension which enables to deploy a test (https://github.com/rmannibucau/batch-tck/commit/18088a4a2445c6a72b2e62f9f1ea583f6057bddb#diff-ba9b558633aa1105e8cdcdbe2e6629fcR13 mock to illustrate it)
2. Extension must be registered in the TCK launcher module (https://github.com/rmannibucau/batch-tck/commit/18088a4a2445c6a72b2e62f9f1ea583f6057bddb#diff-dac6f2888f0eec4583ed6041cf59cbaaR1 - did it in the tck project but in practise it would be in the vendor project)


This does not prevent us to put tags on them (@Tag, it is like categories for previous junit) to mark them as being EE, standalone, CDI, etc and then let the launcher module have multiple executions or so.
But high level this enables to write test once, run them in all possible environments without having to do anything not standard in tests themself.

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


Le ven. 20 mars 2020 à 01:57, Reza Rahman <reza_rahman@xxxxxxxxx> a écrit :

100% sensible. I think most contributors will understand if we just want to focus on Jakarta EE 9 right now and only do other stuff if it is not a distraction. If needed, I think we can easily redirect people to other specifications that also probably need Jakarta EE 9 execution help right now. So the real question is, can we productively get Nishant and other such contributors engaged with the Jakarta EE 9 effort or something less intrusive than the TCK technology choice?

By the way, I appreciate the candor, focus and transparency. I am sure others do too.

Reza Rahman
Jakarta EE Ambassador, Author, Speaker, Blogger

Please note views expressed here are my own as an individual community member and do not reflect the views of my employer.

On 3/19/2020 8:20 PM, Kevin Sutter wrote:
Since you asked, Reza, I'll pipe in...  :-)  Of course, I've been talking with Scott offline about this discussion and the work required for Batch in general.  Although I would like to figure out ways of getting more people involved with Batch, I would caution against spending too many cycles on this TCK question when there is still so much to do for the Jakarta EE 9 deliverable in general.  Specific for Batch, we still need the Specification completed.  As Scott's most recent reply outlined, we still need the TCK to be updated and ready to be plugged into the Platform TCK.  And, we need a Compatible Implementation available somewhere for testing purposes.  In my mind, all of these items are more important than attempting to figure out whether a standalone TCK should be using Arquillian or JUnit5 or some other framework.  Once we start to get a handle on the completion of the Spec, API, TCK, and CI, then we can go back to this question of how to make a standalone TCK workable.

That's my view.  But, of course, I do have a larger goal in mind as the Jakarta EE 9 co-release lead...  :-)

---------------------------------------------------
Kevin Sutter
STSM, MicroProfile and Jakarta EE architect @ IBM
e-mail:  sutter@xxxxxxxxxx     Twitter:  @kwsutter
phone: tl-553-3620 (office), 507-253-3620 (office)    
LinkedIn: https://www.linkedin.com/in/kevinwsutter



From:        Reza Rahman <reza_rahman@xxxxxxxxx>
To:        jakartabatch developer discussions <jakartabatch-dev@xxxxxxxxxxx>
Date:        03/19/2020 18:52
Subject:        [EXTERNAL] Re: [jakartabatch-dev] TCK directions / form factor , e.g. Arquillian - opinions wanted
Sent by:        jakartabatch-dev-bounces@xxxxxxxxxxx



To be honest I would like to hear what some of the others folks here think so far. I count 33 subscribers to this list and we have hardly even had 10 opinions. Surely other people have some thoughts on how Jakarta EE project TCKs should look like? Seems to me like a pretty important topic?

Reza Rahman
Jakarta EE Ambassador, Author, Speaker, Blogger

Please note views expressed here are my own as an individual community member and do not reflect the views of my employer.

On 3/19/2020 7:36 PM, Scott Kurz wrote:
You can see me taking some time to get my head around this thread. I'm realize one area in which I misspoke and possibly misled was using terms like "SE mode".

I also should emphasize this is a post-Jakarta EE 9 discussion ! I'm trying to set us up to work on new function, but for EE 9 we will go with the status quo described below.
---

OK, so a better way to think of what we have right now, in Jakarta EE 8, is that we allow certification in one of two ways:

1. Platform TCK - The batch tests within the Platform TCK (https://github.com/eclipse-ee4j/jakartaee-tck/):
2. Standalone TCK - Built from this batch-tck project, (available at https://projects.eclipse.org/projects/ee4j.batch/downloads)


Problems with the two-TCK status quo:

The batch-tck copies of the test are the true "upstream". This is the repo that holds the issues and PRs that we primarily discuss on the mailing list.

I'm not sure if everyone realizes this, (since this was previously done by IBM and handed directly to Oracle/CTS), we actually do a find/replace operation to transform these tests into the Platform TCK format (this was previously Oracle's "CTS" ). The "transform" complicates debug; one has to understand the transform to, say, work backwards to a batch-tck change. We also need to limit to the common denominator of what's available in Platform TCK (that's why we wrote our own assert util instead of just using JUnit assert, say).

In theory some of this could be mitigated or improved by engaging with the Platform TCK and making it more user-friendly (for our needs). There's a non-trivial learning curve there, this project has its own conventions and features and you'd possibly have to convince other components to accept change they don't see the need for.

So I'm starting from the perspective of wanting to get rid of the Platform TCK, and to make our Standalone TCK usable enough that app servers, e.g. Glassfish, Open Liberty, can easily certify against our standalone TCK.

Problems with the current standalone TCK:

The current "plug points" built into our standalone TCK only include the JobOperator impl itself (via the BatchRuntime's use of ServiceLoader) and a "waiter" to allow strategies other than polling, the default.
There is no plug point for deployment, or even for "setup". A batch implementation like Open Liberty (which I work on) is not "embeddable" in the integration test JVM. To bridge the execution of the test jobs and execute them in a separate app server JVM would require some custom code, unique to each server.


Ideas:

1. Arquillian (my original idea)

Again, I haven't really worked with Arquillian but the problems with our standalone TCK seem to be exactly what Arquillian can solve, and this seems to be the approach existing MicroProfile TCKs and a few EE TCKs are using.
We could use existing adapters/extensions, and stick to proven paths (IIUC, MicroProfile TCKs use TestNG rather than JUnit to integrate with Arquillian better).

One other point that's been raised: I'm not sure how we recreate the "vehicles" of Platform TCK... e.g. run test within a servlet context vs. running within an EJB context. (Though the batch spec says basically nothing about one EE context vs. another, so doesn't seem too valuable, but perhaps could be one day possibly?)

2. JUnit 5 (Romain)

I can appreciate some of Romain's points and criticisms of Arquillian, but after reading that a couple times, I'm still not sure what he's suggesting looks like. Romain, would you be interested in prototyping or show a proof of concept for a couple tests so that we could understand and evaluate?

3. Custom

We could do something altogether different. What if our "deliverable" includes a WAR, within the test zip, packaging the same tests in a different way. The instructions for the WAR say: "go create these two datasources pointing to these two DB tables and set up your job repository however you want. "

I think there's a case to be made that this is a valid TCK. The problem is, someone certifying against the platform doesn't want 20 completely different TCKs, with completely different setup steps. If two types are OK, I guess a case could be made for three, but this really sets us up to be an outlier from the overall platform perspective..a problem child :)
---

Since Romain and Cheng both spoke up against Arquillian, I'm not going to suggest anyone prototype that approach..until we give a chance for other suggestions. But I do think we need some kind of POC of another approach to take the next step and move the conversation forward.

Thanks,
------------------------------------------------------
Scott Kurz
WebSphere Batch and Developer Experience
skurz@xxxxxxxxxx
--------------------------------------------------------

Inactive hide details for Romain Manni-Bucau ---03/18/2020 04:39:44 PM---Some comments inline Le mer. 18 mars 2020 à 21:28, rezRomain Manni-Bucau ---03/18/2020 04:39:44 PM---Some comments inline Le mer. 18 mars 2020 à 21:28, reza_rahman <reza_rahman@xxxxxxxxx>a écrit :

From: Romain Manni-Bucau <rmannibucau@xxxxxxxxx>
To: jakartabatch developer discussions <jakartabatch-dev@xxxxxxxxxxx>
Date: 03/18/2020 04:39 PM
Subject: [EXTERNAL] Re: [jakartabatch-dev] TCK directions / form factor , e.g. Arquillian - opinions wanted
Sent by: jakartabatch-dev-bounces@xxxxxxxxxxx



Some comments inline

Le mer. 18 mars 2020 à 21:28, reza_rahman <reza_rahman@xxxxxxxxx> a écrit :
My suggestion is that we engage the Arquillian team here. While Arquillian does have weaknesses, I think adoption into Jakarta EE TCKs should be important enough such that the development team should at least commit to working with us to solve issues together. I imagine you can easily reach that team but if you need help just let me know.

Hmm, they were not planning junit5 support some months ago "by design", not sure it changes and quarkus to quote only this one tends to validate that for future.

We must keep the opening and stay friendly to people not used to ee IMHO.

No tool is a golden hammer. I don't think we need to adopt Arquillian for every test necessary. However, in cases where it is clearly helpful creating yet another one off solution with likely it's own problems does not make much sense to me - for this project or a given Java EE enterprise application.

Why would you create a new one without Arquillian ;)?
All servers are getting junit5 integrations just to be up to date + adding extensions dynamically is quite easy so no need of anything new for JBatch.
Just plain junit5.

It will also enable to open more tck rather than keeping using a specific heavy stack imho.

Maybe next step is to identify tests requiring "inline" packaging (it is the only arquillian feature needing/providing an api, all others are not needed *in tck* and specs).
Guess none are in jbatch stack - or none cant be rewritten to not need it.

Reza Rahman
Jakarta EE Ambassador, Author, Blogger, Speaker

Please note views expressed here are my own as an individual community member and do not reflect the views of my employer.

Sent via the Samsung Galaxy S7, an AT&T 4G LTE smartphone


-------- Original message --------
From: Scott Kurz <skurz@xxxxxxxxxx>
Date: 3/18/20 7:58 AM (GMT-05:00)
To: jakartabatch developer discussions <jakartabatch-dev@xxxxxxxxxxx>
Subject: Re: [jakartabatch-dev] TCK directions / form factor , e.g. Arquillian - opinions wanted
Romain,

Forgive me if I repeat myself, it's not to drown out your argument, but to say it differently to try to understand.
...

So backing up, in suggesting Arquillian, I have in mind a developer new to the Jakarta Batch. Maybe they ask a question about some behavior on the mailing list, we discuss, and then say, "we should add a TCK test to clarify, would you like to volunteer to write that test?".

At this point, it's too much of a hurdle to expect that they learn the details of the Jakarta EE platform TCK in order to verify their test. Arquillian, even with its flaws, is something that's out there that they can find examples of, and, IIUC, there are existing "adapters" (I'm not using correct terminology) for app server products and also SE+CDI "mode",

Now, if the flaws are great enough, and you're raising some concerns, this is a moot point because Arquillian just doesn't "work" correctly. But trying to understand where the trouble lies... people use Arquillian... I would say the Batch TCK is relatively simple, deploy once, run, assert/validate. Sure we have some timing issues and we poll a lot but I see that would apply in any solution.

Yes, to the person with the role of certifying the entire app server against the entire platform TCK, we have potentially added more work. By pulling Batch out of the Jakarta EE platform TCK, we've created one more "special" TCK ("standalone" if that term isn't too confusing/overloaded). But I'm OK with that because: 1) This is a more specialized role.. not too many people will do this, maybe one per product.. and this person already may have to deal with some quirks of each API within the platform TCK 2) We're not branching out in 20 different directions, just two: Arquillian vs. Jakarta TCK...hopefully there's some commonality / reuse among the set of Arquillian TCKs.

---

That all said, I don't think I really understand what you're proposing though.

If I had to guess, I'd say you are proposing that we continue to develop in duplicate for both the platform TCK https://github.com/eclipse-ee4j/jakartaee-tck/tree/master/src/com/ibm/jbatch/tck
and our individual TCK: https://github.com/eclipse-ee4j/batch-tck/and we look to use ideas like the 'owb-arquillian-standalone' to allow our individual TCK to run in more of an EE environment, for impls that find that convenient?

If so, my objection is that we have to make source updates twice, plus the ultimate validation of these updates requires working with the complicated Jakarta EE platform TCK.

Thanks,
------------------------------------------------------
Scott Kurz
WebSphere Batch and Developer Experience
skurz@xxxxxxxxxx
--------------------------------------------------------

Romain Manni-Bucau ---03/18/2020 02:45:30 AM---Hmm, think I see reality differently Reza, let me expand why: 1. Arquillian was de facto standard fo

From: Romain Manni-Bucau <rmannibucau@xxxxxxxxx>
To: jakartabatch developer discussions <jakartabatch-dev@xxxxxxxxxxx>
Date: 03/18/2020 02:45 AM
Subject: [EXTERNAL] Re: [jakartabatch-dev] TCK directions / form factor , e.g. Arquillian - opinions wanted
Sent by: jakartabatch-dev-bounces@xxxxxxxxxxx



Hmm, think I see reality differently Reza, let me expand why:

1. Arquillian was de facto standard for EE testing something like 5-7 years ago but it kind of shut down. Today it is poorly maintained and the junit 5 big bang helped a lot to make it abandonned by users. All vendors also slowly move to it.
2. EE tck never fully migrated to arquillian except bval, cdi and a few others. Most of the dozens of thousands tests are still "ee custom".

The strength of arquillian is no more its ecosystem - which is often inconsistent and not well compatible - or test facing api but its container abstraction only - and this is usable standalone. We could envision to use it directly but it brings very few for JBatch and most important point for me is to stay smooth for ee consumers. If they want to consume tck they will mainly integrate jakarta tck so better to not force to do an arquillian solution which is often broken in embedded/standalone mode (currznt way to deploy, even EE).

@Scott: I was thinking to this kind of config https://github.com/apache/openwebbeans/blob/master/webbeans-arquillian/owb-arquillian-standalone/src/main/java/org/apache/webbeans/arquillian/standalone/OwbStandaloneConfiguration.java#L37which makes arquillian vs normal runs different.

JSON-P/B tck - for ex - had the advantage to have a process per test and just require to have the impl in the classpath so execution was representative even in standalone.

Now back to JBatch, what do we need?
CDI? Be able to have a CDI container with some beans and not some other? SeContainer is about that and is trivially backable by arquillian container impl if *some vendors* want. Same applies for the "auto war deployment", vendors can do it with arquillian just implementing a custom deployer filtering the classpath from their client libs.

So at the end I think JBatch should stay standalone and CDI centric, even for its tests.





Le mer. 18 mars 2020 à 05:00, Reza Rahman <reza_rahman@xxxxxxxxx> a écrit :
I really prefer we not do anything one off and go the Arquillian approach. The biggest benefit to maintaining uniformity is that contributors across Jakarta EE technologies will not need to re-learn the same thing.

The reality also is that Arquillian remains the de-facto testing solution for Jakarta EE. It is unfortunately still far from perfect but workable. I would like to believe we could adopt Arquillian across Jakarta EE and help smooth out the wrinkles in Arquillian itself in the process.

Reza Rahman
Jakarta EE Ambassador, Author, Speaker, Blogger

Please note views expressed here are my own as an individual community member and do not reflect the views of my employer.

On 3/17/2020 2:31 PM, Scott Kurz wrote:
So one suggestion in the https://github.com/eclipse-ee4j/batch-api/wiki/Ways-To-Contributewas the idea that we could/should convert the TCK to Arquillian (like the JSON-B TCK did).
Let me just back up and walk through my thought process, since I know there was no discussion about that.

And I'd like to solicit opinions about what to do
-----

So right now, we have our tests duplicated across both:
- our individual TCK repo: https://github.com/eclipse-ee4j/batch-tck/This runs test in an SE, non-"app server" environment.
- the platform TCK repo: https://github.com/eclipse-ee4j/jakartaee-tck/Runs tests in an EE, "app server" env.

Obviously we want to source from a single place. Still, we're not forced to use Arquillian, we could do our own thing here..

We could explore:
1. Integrating differently, better with the https://github.com/eclipse-ee4j/jakartaee-tck/project
2. building our own, custom EE standalone TCK

The problems I see with 1.):
- there's a learning curve with the jakartaee-tck as it's its own, custom, kind of complex thing
- how do we keep our "SE" form factor? Not all impls can/will leverage this but some may find it quicker and advantageous to execute tests in this "mode". For batch 1.0 we actually did an Ant transform (find/replace), which seems kind of primitive.

As far as 2., well, it wouldn't be too hard to have a second project take all the test classes, packaging them in a WAR, and putting some sort of JUnit and/or Test NG suite, "wrapping" them.
That would preserve the existing SE mode, and enable an EE mode that avoids the jakartaee-tck complexities.

But from the overall platform perspective, there's value in keeping things simpler and having a smaller number of different directions among the individual component projects. It makes it that much easier to certify against the whole platform.

So if we take for granted that the JSON-B, and beanValidation TCKs have proven the concept of and provided a pattern for a relatively simple Arquillian-based TCK, and that it faciliates SE, non-app server execution mode (which I admit I haven't looked into in detail, not being too familiar w/ Arquillian), then, rather than going off in a new direction (that we don't have yet), I'd like to try to follow these TCKs, and make a second group or pattern of Jakarta TCKs using Arquillian.

Any thoughts? Things I'm missing?

Thanks,
------------------------------------------------------
Scott Kurz
WebSphere Batch and Developer Experience
skurz@xxxxxxxxxx
--------------------------------------------------------

_______________________________________________
jakartabatch-dev mailing list
jakartabatch-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartabatch-dev

_______________________________________________
jakartabatch-dev mailing list
jakartabatch-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartabatch-dev_______________________________________________
jakartabatch-dev mailing list
jakartabatch-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartabatch-dev


_______________________________________________
jakartabatch-dev mailing list
jakartabatch-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartabatch-dev_______________________________________________
jakartabatch-dev mailing list
jakartabatch-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartabatch-dev




_______________________________________________
jakartabatch-dev mailing list
jakartabatch-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartabatch-dev

_______________________________________________
jakartabatch-dev mailing list
jakartabatch-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartabatch-dev




_______________________________________________
jakartabatch-dev mailing list
jakartabatch-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartabatch-dev


_______________________________________________
jakartabatch-dev mailing list
jakartabatch-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartabatch-dev
_______________________________________________
jakartabatch-dev mailing list
jakartabatch-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartabatch-dev

Back to the top