Re: [platform-releng-dev] again on session tests

[Jeff McAffer <Jeff_McAffer@xxxxxxxxxx>]

Mechanically this seems fine. It is
basically using the mechanisms that we have already.  There is a question
as to how people want to organize tests.  Using the support outlined
here, each method in a TestCase class will spawn a different VM.  This
has good (+) and bad (-) points.  
+ simple
+ *may* be natural in the true Junit
philosophy (I confess to not fully getting it)
- makes writing/composing complex test
"scenarios" harder as you do not get to use the TestSuite and
TestCase grouping/ordering mechanisms (e.g., if you write a startup test,
a CVS check out test and a full build test, you cannot somehow then group
these into a suite without forcing multiple VMs).
- may make it hard to talk about "scenarios"
as a scenario would then just map onto a particuarl method on a TestCase
class.

As for the mechanics, depending on PDE
is a bummer for this.  We should seek to push that function down or
duplicate.  Don't we also have something like this to run the test
suites in releng mode?

Jeff

Rafael Chaves/Ottawa/IBM@IBMCA
Sent by: platform-releng-dev-admin@xxxxxxxxxxx

09/08/2004 11:33 AM

Please respond to
platform-releng-dev

To	platform-releng-dev@xxxxxxxxxxx
cc
Subject	[platform-releng-dev] again on session tests

Platform/Core has been doing some work on running JUnit tests that end
up
spawning a second instance of Eclipse, or "session tests". There
is a lot
of scenarios that can only be tested with session tests, as startup
performance tests.

We have been doing some work in that area, and were curious if others had
the same need, and what approach was (or is planned to be) used to support
that.

Here is the approach we are taking:
- session tests use a special SessionTestSuite class to build their test
suites
- session tests are actually just proxies for the actual tests that know
which plug-in/class/method implements the test case
- when a session test is run, it launches a second instance of Eclipse
to
run a application that is/has a test runner. This test runner will run
that
single test case and report back the test result to the original instance
using a socket (right now we use org.eclipse.pde.junit.runtime test
applications and protocol for doing that, but it causes an undesirable
dependency on a non-platform plug-in). Once the test case has finished,
this second Eclipse instance terminates
- back to the original instance, we check whether the test failed/passed.
If it failed, we fake a corresponding error/failure to occur at the proxy
test, so it is reported as a regular test error/failure to the JUnit
framework

This way, session tests don't need any special support other than using
the
SessionTestSuite class when building their test suites. Also, the current
support for running automated tests as part of the RelEng test framework
or
by using the PDE/JDT support in the SDK still works.

Any feedback is welcome.

Rafael

_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev