Re: [platform-releng-dev] again on session tests

[Christof Marti <christof_marti@xxxxxxxxxx>]

What is the reason for using a second instance of eclipse? Is the overhead
from the automated testing framework too big when registering a session
test directly in the test.xml?

Christof


Wednesday, September 08, 2004 10:55 PM
To: platform-releng-dev@xxxxxxxxxxx
cc:
From: Nick Edgar <Nick_Edgar@xxxxxxxxxx>
Subject: Re: [platform-releng-dev] again on session tests



Note that the Workbench has a very simple handshaking protocol for running
the tests once the workbench is ready.  There is currently no support for
inter-process communication though.
See ITestHarness and TestableObject.  The design goals behind this (small)
API were to provide maximum decoupling between the test harness, the
workbench, and the RCP app.  For example, the RCP app doesn't need to be
aware of the test harness at all, and the workbench does not need to know
about the PDE test runner.  Ideally the PDE test runner wouldn't have to
know about the workbench either, but it currently needs to obtain the
TestableObject from PlatformUI, so there's a dependency there.
Although these types currently exist at workbench level, they have no
dependencies on other workbench types.  This was done in the hopes that
they could eventually be generalized and pushed down.   Maybe now is the
time.

The control flow is:
- eclipse runs the test harness app (e.g. the PDE test runner), telling it
which main app to run (e.g. the IDE)
- the test harness sets itself as the ITestHarness implementation on the
workbench's TestableObject:
PlatformUI.getTestableObject().setTestHarness(this)
- the test harness runs the main app (an RCP app, of course)
- the RCP app creates and runs the workbench as usual
(PlatformUI.createAndRunWorkbench(Display))
- the workbench initializes itself, restores any previously saved state,
etc.
- just before the workbench is about to run the event loop, it notifies
the test harness that it can run the tests (ITestHarness.runTests())
  - this is actually done in a forked thread
  - the test harness tells the TestableObject that it is about to start
  testing (TestableObject.testingStarting())
  - this allows the workbench to put itself into test mode, e.g. make sure
  error dialogs don't pop up and hang the tests
  - the test harness tells the TestableObject to run the tests
  (TestableObject.runTest(Runnable))
  - currently, due to difficulty in breaking up the flow of control of
  JUnit tests organized in a suite, all the tests are run in the same
  runnable
  - the workbench's TestableObject runs the tests in an asyncExec, since
  the UI tests expect to be run in the UI thread
  - since runTests() was called from a forked thread, the workbench fell
  through to processing UI events, so it will process the asyncExec when
  there are no more OS events
  - the test harness tells the TestableObject that it is done testing
  (TestableObject.testingFinished())
  - the workbench reverts back to non-test mode

Does this in any way map onto the kind of inter-process communication
you're doing now?
Thoughts?

Nick




Rafael Chaves/Ottawa/IBM@IBMCA
Sent by: platform-releng-dev-admin@xxxxxxxxxxx
09/08/2004 11:33 AM
Please respond to
platform-releng-dev


To
platform-releng-dev@xxxxxxxxxxx
cc

Subject
[platform-releng-dev] again on session tests










Platform/Core has been doing some work on running JUnit tests that end up
spawning a second instance of Eclipse, or "session tests". There is a lot
of scenarios that can only be tested with session tests, as startup
performance tests.

We have been doing some work in that area, and were curious if others had
the same need, and what approach was (or is planned to be) used to support
that.

Here is the approach we are taking:
- session tests use a special SessionTestSuite class to build their test
suites
- session tests are actually just proxies for the actual tests that know
which plug-in/class/method implements the test case
- when a session test is run, it launches a second instance of Eclipse to
run a application that is/has a test runner. This test runner will run
that
single test case and report back the test result to the original instance
using a socket (right now we use org.eclipse.pde.junit.runtime test
applications and protocol for doing that, but it causes an undesirable
dependency on a non-platform plug-in). Once the test case has finished,
this second Eclipse instance terminates
- back to the original instance, we check whether the test failed/passed.
If it failed, we fake a corresponding error/failure to occur at the proxy
test, so it is reported as a regular test error/failure to the JUnit
framework

This way, session tests don't need any special support other than using
the
SessionTestSuite class when building their test suites. Also, the current
support for running automated tests as part of the RelEng test framework
or
by using the PDE/JDT support in the SDK still works.

Any feedback is welcome.

Rafael

_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev


_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
 http://dev.eclipse.org/mailman/listinfo/platform-releng-dev