Re: [platform-releng-dev] again on session tests

[Thomas Watson <tjwatson@xxxxxxxxxx>]

Does the second instance of eclipse
use the same configuration directory as the first instance eclipse?  I
may be able to work around the fact that the same set of bundles is installed
in both instances of eclipse.  I could using the SessionTestSuite
to install and start the "director" bundle on the first instance
of eclipse and then the SessionTestSuite could launch the second instance
of eclipse and use a "target" application that would install
the target bundle on the second instance of eclipse.   To do this
it would require that each instance of eclipse is launched using its own
configuration directory.  The reason is the "target" will
be installing/uninstalling/updating many bundles to the "target"
configuration.  I think this would cause problems if both instances
shared the same configuration.

Thomas Watson

Rafael Chaves <Rafael_Chaves@xxxxxxxxxx>
Sent by: platform-releng-dev-admin@xxxxxxxxxxx

09/09/2004 10:29 AM

Please respond to
platform-releng-dev

To	platform-releng-dev@xxxxxxxxxxx
cc
Subject	Re: [platform-releng-dev] again on session tests

The need for building/specifying a new configuration when running the target
is a real requirement, but one that has not been addressed yet (right now
the director's configuration's is used when launching the target). It is
on the radar, though. Suggestions on how to tackle this problem are welcome.

Rafael

Thomas Watson <tjwatson@xxxxxxxxxx>
Sent by: platform-releng-dev-admin@xxxxxxxxxxx

09/09/2004 09:00 AM

Please respond to
platform-releng-dev

To	platform-releng-dev@xxxxxxxxxxx
cc
Subject	Re: [platform-releng-dev] again on session tests

There is existing set of test cases that are used to test the OSGi specification.
 This test suite is developed and published by OSGi.  Unfortunately
it is not based on JUnit.  To run the OSGi TCK you are required to
launch two instances of the OSGi Framework.  One that acts as the
"director" and the other acts as the "target".  The
director pushes a testcase to the target, then the target runs the testcase
and reports back to the director the results. 

I'm not suggesting that we use this model.  Instead I'm trying to
determine if I can use your concept of a session test to launch the second
instance of eclipse.  In this case the director would be launched
with the first instance of eclipse and then the target would be launched
by the second instance.  Would something like this be possible?  Are
you able to specify a different set of bundles to be installed on the second
instance that is launched or do the two instances have to have the same
set of bundles? 

Thomas Watson

Jeff McAffer <Jeff_McAffer@xxxxxxxxxx>
Sent by: platform-releng-dev-admin@xxxxxxxxxxx

09/08/2004 10:23 PM

Please respond to
platform-releng-dev

To	platform-releng-dev@xxxxxxxxxxx
cc
Subject	Re: [platform-releng-dev] again on session tests

Yes, we have the work item to do startup tests.  They should be possible
in the very near future. (e.g., tomorrow).  The main issue here (as
I see it) is that all tests inherently run in a session.  If we want
to have control over these we need to be able to control how they are run
(e.g., the command line used to launch Eclipse).  

We have seen elsewhere the notion of "scenario" and "variations"
where the scenario was the basic function operations to do and the variations
could be anything from what VM to use to different command line (e.g.,
VM) args to what workspace to use etc.  The discussion below has bearing
because it implies that scenario = method on TestCase since a new VM is
launched for each test* method.  See my other note for comments on
whether or not this is an optimal mapping.   

An alternative is to launch a VM for every TestSuite.  This allows
composition of Tests to build a scenario. That may or may not be desirable/natural.

Jeff

Nick Edgar/Ottawa/IBM@IBMCA
Sent by: platform-releng-dev-admin@xxxxxxxxxxx

09/08/2004 04:25 PM

Please respond to
platform-releng-dev

To	platform-releng-dev@xxxxxxxxxxx
cc
Subject	Re: [platform-releng-dev] again on session tests

The UI team wants to have similar session tests, e.g. time to bring up
a 
simple RCP app, or time to bring up the IDE.
We have held of investigating our own solution for this in the hopes that
you would provide an answer <g>.

Nick




Rafael Chaves/Ottawa/IBM@IBMCA 
Sent by: platform-releng-dev-admin@xxxxxxxxxxx
09/08/2004 11:33 AM
Please respond to
platform-releng-dev


To
platform-releng-dev@xxxxxxxxxxx
cc

Subject
[platform-releng-dev] again on session tests










Platform/Core has been doing some work on running JUnit tests that end
up
spawning a second instance of Eclipse, or "session tests". There
is a lot
of scenarios that can only be tested with session tests, as startup
performance tests.

We have been doing some work in that area, and were curious if others had
the same need, and what approach was (or is planned to be) used to support
that.

Here is the approach we are taking:
- session tests use a special SessionTestSuite class to build their test
suites
- session tests are actually just proxies for the actual tests that know
which plug-in/class/method implements the test case
- when a session test is run, it launches a second instance of Eclipse
to
run a application that is/has a test runner. This test runner will run
that
single test case and report back the test result to the original instance
using a socket (right now we use org.eclipse.pde.junit.runtime test
applications and protocol for doing that, but it causes an undesirable
dependency on a non-platform plug-in). Once the test case has finished,
this second Eclipse instance terminates
- back to the original instance, we check whether the test failed/passed.
If it failed, we fake a corresponding error/failure to occur at the proxy
test, so it is reported as a regular test error/failure to the JUnit
framework

This way, session tests don't need any special support other than using
the
SessionTestSuite class when building their test suites. Also, the current
support for running automated tests as part of the RelEng test framework
or
by using the PDE/JDT support in the SDK still works.

Any feedback is welcome.

Rafael

_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev


_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev