One question around this that will mostly affect full EE vendors. With the old way of running the TCK most of the tests were handled via a single adaptor/test harness (with a few exceptions such as CDI). Once you had your TCK setup you could for the most part run any subset of the tests using the same configuration.
The current underlying problem is that the
Java Test Harness + Apache Ant + Platform TCK are not that well understood, so the number of contributors adding/updating TCK tests is too low. I wouldn't expect that TCKs will use JUinit + Arquillian forever. Hopefully there will be less complexity with the new TCKs which should make it easier to later switch to the latest build + unit test libraries.
If all the TCKs are going to be split up my concern is that over time the requirements of the test harness for each project may diverge, and we end up with vendors needing to maintain multiple versions of kinda similar but subtly different test adaptors for each TCK.
Platform requirements should still be validated in a Platform/profile TCK. The Platform TCK could extend the Servlet TCK tests with some (platform/profile level) cost to make changes for each release as needed. The big payback on the effort is that TCK tests will be easier to maintain which improves the overall Jakarta EE development process.
Do you think this will be a real concern?
The EE 8 - 9 upgrades were pretty smooth sailing but I recall there being work involved with earlier EE releases with a lot of TCK configuration changes and resolving several TCK failures as well. I vaguely recall the first TCK releases as well that required "make" tools from four separate vendors (at least they did on Microsoft Windows). The make scripts were rewritten to use Apache Ant and that has lasted a really long time.
I was concerned as well when I first read the
https://github.com/jakartaee/platform-tck/issues/51 issue about converting the Platform TCK into multi-module Maven projects but I think it is the best approach to take until we have new requirements to do something else in a later release.
I have not looked in depth at how this is being implemented so I am not really sure. In an ideal world IMHO as much as possible of the test infrastructure code should still be shared but I am not sure if it is being done that way.
I think it is a healthy thing to eliminate technical debt while rewriting a huge code base such that we are only left with needed features and will be better prepared for changes in the future. We wanted to do this for EE 10 but decided to wait for EE 11 and here we are. :-)