The idea of using JUnit 5 as a replacement of Arquillian is tempting. I personally would like that Arquillian itself supported JUnit 5 natively with a JUnit extension so that it can just be simply added to any JUnit 5 test to run it in a container. But we don't need to introduce JUnit 5 in a single step.
Over the weekend I had a deeper look at both the standalone TCK and the Batch part in the Jakarta CTS.
After thinking about it for a while, I suggest the following, for the reasons explained below:
1. convert the existing Test NG tests into Arquillian tests in such a way, that we abstract the Arquillian-specific code into abstract test classes that all other tests would extend. This should be simple to do and would require negligible refactoring. We can create an example project to run SE implementations that will run JBatch using the Weld Arquillian container. After this, we could discard the tests in the Jakarta CTS and use the same test suite for both SE and EE.
2. explore how we could refactor to using JUnit 5. If we find a good solution, it would then be possible to achieve all the advantages of Romain's suggestion and also provide an example for other projects how they can modernize their TCKs with JUnit 5. It could also provide a good feedback for the Arquillian JUnit 5 integration that could help improve the Arquillian project.
The tests are written in Test NG and a conversion to Arquillian + Test NG would solve how to run them in EE, outside of the Jakarta CTS. Exactly what Bean Validation and CDI TCKs do now. However, running non-EE implementations would be more complicated as now and require configuring either a Java SE or Weld container. There's also a problem of packaging, as Arquillian requires a deployment package, which is a WAR or EAR in EE but usually plain JAR in SE. This is also an issue we faced in MicroProfile TCKs where most of the TCKs create a WAR package and any SE implementation needs to create an Arquillian extension that repackages it to JAR before running the tests.
On the other hand, rewriting the tests to JUnit 5 as Romain suggests, allows more flexibility and make it easier to develop new tests. The Batch TCK doesn't need much from from Arquillian, only packaging (which is a bit flawed as I explained), deployment and running in CDI environment. This can be supported in a different way. However, as I assumed, Romain's example assumes a lot of work on the integration side, would require the
InvocationInterceptor feature available since Junit 5.5 and most EE integrations would still most probably rely on their Arquillian connectors rather than writing an JUnit 5 extension from scratch (I really didn't see any direct JUnit 5 support for any app server yet despite that Romain says that "all servers are getting junit 5 integrations" ). We could mitigate it if we provided e.g. some helper classes in the TCK to simplify the integration.
For those who didn't get Romain's idea with JUnit 5, it's basically using JUnit 5 with its extension points instead of Arquillian. Integrators would write a JUnit 5 extension that would do what server connectors do in Arquillian. So they could be just a thin wrapper over an existing Arquillian connector but SE implementations would just need a very simple extension to configure the impl or even no extension at all. But a JUnit 5 extension only gets general meta information about the test. The TCK would probably need to provide some information via JUnit tags or helper methods to inform which classes and resources to pack.
Ondro