[wtp-releng] Status of JUnit tests (and framework)

[David M Williams <david_williams@xxxxxxxxxx>]

Those that work closely with me will be glad to hear
the 
test framework is now working well enough that I stopped

to take a shower and shave :)

But, seriously, it is working fairly well.
Here's some nitty gritty in bug reports ... I'm not
sure if the
two platform bugs will turn out to be bugs, or intentional
features.

        https://bugs.eclipse.org/bugs/show_bug.cgi?id=176896
        https://bugs.eclipse.org/bugs/show_bug.cgi?id=178163
        https://bugs.eclipse.org/bugs/show_bug.cgi?id=177835

Part of the outcome of this churn is that I think
we will be better off
having our own lightweight framework, instead of trying
to fit in
to someone else. I'm not sure we gain enough re-use
to make it
worthwhile. We will continue to re-use some of it
... the core Eclipse
applications ... but the ant scripts can be greatly
simplified. 


In the latest build, I did see 4 "did not finish"
tests:

        org.eclipse.jst.j2ee.tests.bvt.AutomatedBVT.
     
        org.eclipse.jst.jsf.core.tests.AllTests.
         
        org.eclipse.jst.jsf.metadata.tests.AllTests.
     
        org.eclipse.jst.ws.tests.unittest.AllWSJUnitTests.

I'll retry upping the time out to 30 minutes (instead
of 10),  but the
AllWSJUnitTests I think is a "real" problem
that existed before.
I'm not sure about the others (but, will be looking
to reduce the
default time limit back to 10 minutes ... unless many
think it
should be higher.

What remains? Still lots of cleanup, in code and web
pages,
some of which will involve cleanup to each test.xml
file ...
if we (or, I) get real ambitious.
but,

Here are a couple items your team should do:

        Check if actual results
match what you expect, in terms of number of tests, etc?

        Especially you that use
"extraVMargs" please check those extra VM args seem to
        have the desired effect.

        We used to "group"
by plugin, but now leave collected per test suite. Are they all showing
up?
        Is there a need to do
"per bundle"?