| Re: [cdi-dev] CDI TCK for Core Profile - test count? |
I’m starting to really think that my test count is the correct one here, Scott. The TestSuite.xml you linked to is full of stuff like this:
<testcase name="testDecoratedTypes" classname="org.jboss.cdi.tck.tests.full.decorators.definition.DecoratorDefinitionTest" time="0.029"/><testcase name="testDecoratorDoesNotImplementDelegateType" classname="org.jboss.cdi.tck.tests.full.decorators.definition.DecoratorDefinitionTest" time="0.024"/>
<testcase name="testDecoratorIsManagedBean" classname="org.jboss.cdi.tck.tests.full.decorators.definition.DecoratorDefinitionTest" time="0.027"/>
<testcase name="testDecoratorOrdering" classname="org.jboss.cdi.tck.tests.full.decorators.definition.DecoratorDefinitionTest" time="0.022"/>
<testcase name="testDelegateInjectionPoint" classname="org.jboss.cdi.tck.tests.full.decorators.definition.DecoratorDefinitionTest" time="0.023"/>
<testcase name="testDuplicateBindingsOnResolveDecoratorsFails" classname="org.jboss.cdi.tck.tests.full.decorators.definition.DecoratorDefinitionTest" time="0.026"/>
<testcase name="testEmptyTypeSetOnResolveDecoratorsFails" classname="org.jboss.cdi.tck.tests.full.decorators.definition.DecoratorDefinitionTest" time="0.025"/>
<testcase name="testInstanceOfDecoratorForEachEnabled" classname="org.jboss.cdi.tck.tests.full.decorators.definition.DecoratorDefinitionTest" time="0.024"/>
<testcase name="testNonBindingsOnResolveDecoratorsFails" classname="org.jboss.cdi.tck.tests.full.decorators.definition.DecoratorDefinitionTest" time="0.022"/>
<testcase name="testNonEnabledDecoratorNotResolved" classname="org.jboss.cdi.tck.tests.full.decorators.definition.DecoratorDefinitionTest" time="0.017"/>
The entire class of DecoratorDefinitionTest (https://github.com/jakartaee/cdi-tck/blob/master/impl/src/main/java/org/jboss/cdi/tck/tests/full/decorators/definition/DecoratorDefinitionTest.java) is marked with CDI_FULL and should not be executed here.
I’ve opened an issue here with an example of our CDI Lite (Core Profile) results -- https://github.com/jakartaredhat/wf-core-tck-runner/issues/1
-Brian
Brian Decker
Jakarta Platform TCK Lead, Open Liberty
Home Office
(612) 216-4881
Roch. Office
t/l 553-4686
bmdecker@xxxxxxxxxx
From: cdi-dev <cdi-dev-bounces@xxxxxxxxxxx> On Behalf Of Scott Stark
Sent: Friday, June 17, 2022 8:22 PM
To: cdi developer discussions <cdi-dev@xxxxxxxxxxx>
Subject: [EXTERNAL] Re: [cdi-dev] CDI TCK for Core Profile - test count?
That is for the web profile run though. They have been asking about the core profile which is only including tests that are valid for CDI Lite. I consistently see 1263 for WildFly/Weld for different run environments. I'll look into the
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
That is for the web profile run though. They have been asking about the core profile which is only including tests that are valid for CDI Lite. I consistently see 1263 for WildFly/Weld for different run environments. I'll look into the logging, but added an example target/surefire-reports/TEST-TestSuite.xml From a run here:
Create an issue with your TEST-TestSuite.xml file so I can see what tests are different:
https://github.com/jakartaredhat/wf-core-tck-runner/issues
One thing I seemed to notice is that it looked test archives are created even if all of the tests associated with the test archive are excluded. It is on my todo list to look into why that is as well.
On Jun 17, 2022 at 7:29:00 PM, arjan tijms <arjan.tijms@xxxxxxxxx> wrote:
Hi,
For GlassFish it's
===============================================
CDI TCK
Total tests run: 1831, Passes: 1831, Failures: 0, Skips: 0
===============================================
You can see our runner and its options here:
Specifically we run all the tests for EE, and only exclude SE.
I'm not sure that it's correct for Jakarta EE certification to exclude "cdi-full'.
Kind regards,
Arjan Tijms
On Sat, Jun 18, 2022 at 12:41 AM Brian M Decker <bmdecker@xxxxxxxxxx> wrote:
Yikes, okay. So we have 3 different numbers so far.
Interestingly, when I change my automation to exclude only se (and not cdi-full), I get a test count of 1267, which is much closer (but still not exactly the same) as what others are seeing. Are y’all seeing a bunch of org.jboss.cdi.tck.tests.full.* executed in your runs? If so, I would wager that your excludes aren’t actually being honored for some reason.
In my run, I see exactly one of those tests run -- org.jboss.cdi.tck.tests.full.lookup.injection.visibility – and that seems like it might be a mistake. I would think that all tests in org.jboss.cdi.tck.tests.full should be tagged with @Test(groups = CDI_FULL).
-Brian
Brian Decker
Jakarta Platform TCK Lead, Open Liberty
Home Office
(612) 216-4881
Roch. Office
t/l 553-4686
bmdecker@xxxxxxxxxx
From: cdi-dev <cdi-dev-bounces@xxxxxxxxxxx> On Behalf Of Alfonso Altamirano
Sent: Friday, June 17, 2022 4:01 PM
To: cdi developer discussions <cdi-dev@xxxxxxxxxxx>
Subject: [EXTERNAL] Re: [cdi-dev] CDI TCK for Core Profile - test count?
On our side we are seeing the following amount of tests with the following core profile configuration: <!--core profile--> <excludedGroups>cdi-full,se</excludedGroups> <dependenciesToScan>
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
ZjQcmQRYFpfptBannerEnd
On our side we are seeing the following amount of tests with the following core profile configuration:
<!--core profile-->
<excludedGroups>cdi-full,se</excludedGroups>
<dependenciesToScan>
<dependency>jakarta.enterprise:cdi-tck-core-impl</dependency>
</dependenciesToScan>
total number of test: 1295
if anyone can confirm the number of tests, that can be good
From: cdi-dev <cdi-dev-bounces@xxxxxxxxxxx> on behalf of Brian M Decker <bmdecker@xxxxxxxxxx>
Sent: Friday, June 17, 2022 3:31 PM
To: cdi developer discussions <cdi-dev@xxxxxxxxxxx>
Subject: [cdi-dev] CDI TCK for Core Profile - test count?
Hello,
I’m attempting to run the CDI TCK for Core Profile and am getting a dramatically different test count than I see posted elsewhere, so I was hoping to compare notes with someone (Scott Stark, perhaps?) on what exactly I might be missing.
The User Guide indicates that I should be running with the cdi-tck-core-impl TCK artifact and excluding groups cdi-full & se. When I do this, my runner executes only 725 tests. For comparison, I can see from results published online that Wildfly is reporting 1263 tests when running in this configuration. This is obviously a pretty massive difference. Unfortunately, the Wildfly output doesn’t show exactly what tests are being run, so I can’t see which specific things are missing. I can provide the breakdown of tests being run on my end upon request.
Thanks.
-Brian
Brian Decker
Jakarta Platform TCK Lead, Open Liberty
Home Office
(612) 216-4881
Roch. Office
t/l 553-4686
bmdecker@xxxxxxxxxx
_______________________________________________
cdi-dev mailing list
cdi-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdi-dev_______________________________________________
cdi-dev mailing list
cdi-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdi-dev_______________________________________________
cdi-dev mailing list
cdi-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/cdi-dev