[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
Re: [platform-releng-dev] again on session tests
|
Some points:
- scripting with Ant is really not much
fun. It is both complex (see John's message) and brittle (have to
modify test.xml files everytime you add a test etc)
- managing this through such scripts
means that user cannot run the tests easily from their workspace using
the JUnit UI.
- the proposed "a way to"
mechanisms are very specific to this situation. What if I want to
do other things differnetly on the last run? or things differently
on the first run as well?
The point here is that Rafael's approach
has session tests be as much like normal tests as possible. That
allows users to code them naturally and take advantage of all the other
mechanisms available. WRT the example below, assuming the tests were
NOT session tests, what is the code pattern for running multiple iterations
of some test and asserting only on the aggregate result? I think
a design goal here should be to make that same pattern work for session
tests.
Jeff
p.s., I called them "patterns"
just for Erich...
Christof Marti/Zurich/IBM@IBMCH
09/09/2004 01:05 PM
|
To
| Rafael Chaves/Ottawa/IBM@IBMCA@IBMDE
|
cc
| Erich Gamma, Andre Weinand,
Jeff McAffer/Ottawa/IBM@IBMCA, Nick Edgar/Ottawa/IBM@IBMCA@IBMDE
|
Subject
| Re: [platform-releng-dev]
again on session tests |
|
For repeated measurements we suggest a solution
where the test is run from test.xml n times. Each run would tag its single
measurement with some tag 'foo' and commit it to the database. Only the
last run's assertPerformance() would take all measurements with that tag
'foo' and compare them against reference measurements.
What is need from the performance plug-in
is:
- a way to tag runs with 'foo' from test.xml
in all n runs
- a way to disable assertPerformance() from
test.xml in the first n-1 runs
- a way to compare measurements taged with
'foo' against a reference in the n-th run (like: assertPerformance(tag),
where the tag would be 'foo' in this example)
This approach would avoid the second instance
of Eclipse and the additional communication support from the second instance
back to the first.
Christof
Thursday, September 09, 2004 5:48 PM
To: Christof Marti/Zurich/IBM@IBMCH
cc:
From: Rafael Chaves/Ottawa/IBM@IBMCA@IBMDE
Subject: Re: [platform-releng-dev] again
on session tests
I am forwarding this directly
to you because it seems we (Ottawa) are not having our Notes messages being
sent to non-Notes destinations (as the platform-releng list).
Forwarding...
September 9, 2004 11:14
AM
To: platform-releng-dev@xxxxxxxxxxx
cc:
From: Rafael Chaves/Ottawa/IBM@IBMCA
Subject: Re: [platform-releng-dev]
again on session tests
We have considered doing it the way
you suggested and the reasons we didn't go for it were:
a) flexibility/easy-of-use: developers
would have to tweak their ant scripts to create a separate target for every
performance test. Integration with PDE/Junit would not work (for most cases).
b) doing the measuring several times
before commiting/asserting would not be possible.
Rafael
Christof Marti <christof_marti@xxxxxxxxxx>
Christof Marti <christof_marti@xxxxxxxxxx>
Sent by: platform-releng-dev-admin@xxxxxxxxxxx
09/09/2004 06:21 AM
Please respond to
platform-releng-dev
To
platform-releng-dev@xxxxxxxxxxx
cc
Subject
Re: [platform-releng-dev] again on session
tests
What is the reason for using a second
instance of eclipse? Is the overhead
from the automated testing framework
too big when registering a session
test directly in the test.xml?
Christof
Wednesday, September 08, 2004 10:55
PM
To: platform-releng-dev@xxxxxxxxxxx
cc:
From: Nick Edgar <Nick_Edgar@xxxxxxxxxx>
Subject: Re: [platform-releng-dev]
again on session tests
Note that the Workbench has a very
simple handshaking protocol for running
the tests once the workbench is ready.
There is currently no support for
inter-process communication though.
See ITestHarness and TestableObject.
The design goals behind this (small)
API were to provide maximum decoupling
between the test harness, the
workbench, and the RCP app. For
example, the RCP app doesn't need to be
aware of the test harness at all, and
the workbench does not need to know
about the PDE test runner. Ideally
the PDE test runner wouldn't have to
know about the workbench either, but
it currently needs to obtain the
TestableObject from PlatformUI, so
there's a dependency there.
Although these types currently exist
at workbench level, they have no
dependencies on other workbench types.
This was done in the hopes that
they could eventually be generalized
and pushed down. Maybe now is the
time.
The control flow is:
- eclipse runs the test harness app
(e.g. the PDE test runner), telling it
which main app to run (e.g. the IDE)
- the test harness sets itself as the
ITestHarness implementation on the
workbench's TestableObject:
PlatformUI.getTestableObject().setTestHarness(this)
- the test harness runs the main app
(an RCP app, of course)
- the RCP app creates and runs the
workbench as usual
(PlatformUI.createAndRunWorkbench(Display))
- the workbench initializes itself,
restores any previously saved state,
etc.
- just before the workbench is about
to run the event loop, it notifies
the test harness that it can run the
tests (ITestHarness.runTests())
- this is actually done in a
forked thread
- the test harness tells the
TestableObject that it is about to start
testing (TestableObject.testingStarting())
- this allows the workbench
to put itself into test mode, e.g. make sure
error dialogs don't pop up and
hang the tests
- the test harness tells the
TestableObject to run the tests
(TestableObject.runTest(Runnable))
- currently, due to difficulty
in breaking up the flow of control of
JUnit tests organized in a suite,
all the tests are run in the same
runnable
- the workbench's TestableObject
runs the tests in an asyncExec, since
the UI tests expect to be run
in the UI thread
- since runTests() was called
from a forked thread, the workbench fell
through to processing UI events,
so it will process the asyncExec when
there are no more OS events
- the test harness tells the
TestableObject that it is done testing
(TestableObject.testingFinished())
- the workbench reverts back
to non-test mode
Does this in any way map onto the kind
of inter-process communication
you're doing now?
Thoughts?
Nick
Rafael Chaves/Ottawa/IBM@IBMCA
Sent by: platform-releng-dev-admin@xxxxxxxxxxx
09/08/2004 11:33 AM
Please respond to
platform-releng-dev
To
platform-releng-dev@xxxxxxxxxxx
cc
Subject
[platform-releng-dev] again on session
tests
Platform/Core has been doing some work
on running JUnit tests that end up
spawning a second instance of Eclipse,
or "session tests". There is a lot
of scenarios that can only be tested
with session tests, as startup
performance tests.
We have been doing some work in that
area, and were curious if others had
the same need, and what approach was
(or is planned to be) used to support
that.
Here is the approach we are taking:
- session tests use a special SessionTestSuite
class to build their test
suites
- session tests are actually just proxies
for the actual tests that know
which plug-in/class/method implements
the test case
- when a session test is run, it launches
a second instance of Eclipse to
run a application that is/has a test
runner. This test runner will run
that
single test case and report back the
test result to the original instance
using a socket (right now we use org.eclipse.pde.junit.runtime
test
applications and protocol for doing
that, but it causes an undesirable
dependency on a non-platform plug-in).
Once the test case has finished,
this second Eclipse instance terminates
- back to the original instance, we
check whether the test failed/passed.
If it failed, we fake a corresponding
error/failure to occur at the proxy
test, so it is reported as a regular
test error/failure to the JUnit
framework
This way, session tests don't need
any special support other than using
the
SessionTestSuite class when building
their test suites. Also, the current
support for running automated tests
as part of the RelEng test framework
or
by using the PDE/JDT support in the
SDK still works.
Any feedback is welcome.
Rafael
_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev
_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev
_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev