[cdt-dev] Some JUnit related stuff - automated testing

[Peter Graves <pgraves@xxxxxxx>]

Hey all,
I have been doing a bit of work with the junit tests relating to report 
generation over the last little while and wanted to get some opinions.

I now have a setup where we can generate html reports similar to those 
that the jdt folks generate at the end of each build, with 2 main extras:

1. It will attempt to pull out a list of all the bugids that the various 
test cases passed over, and separate them by the test outcome (pass, 
fail, error, expected fail, and unexpected pass).
There are 2 ways I try to get bugzilla numbers out of test cases:
-The test case name.  If it starts with testBug<bugid> (i.e. 
testBug38543BadString) I can pull that out.
-The failure message.  If the failure message starts with PR:<bugid> 
(i.e.  "PR:38654 Binariy.getCPU returns wrong string") I can pull that out.

2. It will try to catch expected failures, and unexpected successes. The 
ways I try to catch these are:
-Test class name.  If the class ends with FailedTests (i.e. 
DOMFailedTests) I assume all tests that succeed are expected fails 
(these are in the failures directory and a have had the success criteria 
reversed).  I also assume that if a test in one of these classes fails, 
that that is really an unexpected success. I am not sure I like this.. 
what do others think?
-Failure message.  If a test fails with a failure message starting with 
PR:<bugid> (as above), then I assume that this is an expected failure 
with a open bugzilla report. I am thinking it may be worth changing this 
so that if a pr is open the message should say XPR:  and once it's fixed 
  and the pr is closed it would change back to just PR so that if a bug 
resurfaced it could be caught. Opinions?
-It's not there right now, but the AutomatedIntegrationSuite adds in 
markers when it builds the suite, which I can key off of as well.

The costs:
If we want the pr lists to be complete we would have to make sure to 
name the test cases properly (this seems to be the case, the only one I 
really noticed that was different was  LokiFailures in the failures 
directory).

To make this work I am using a custom Junit results formatter that the 
org.eclipse.test plugin has to be able to load. To do this we would need 
a new plugin fragment that linked to org.eclipse.test, that the results 
formatter would go in.  We could also use this fragment to store the 
test automation framework stuff (scripts, property files, etc) and any 
common test code.

Comments?

Thanks
 -Peter