[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [platform-releng-dev] again on session tests
|
Ant most definitely does NOT support
looping of any kind. It barely (and I mean barely) supports conditional
control flow. Ant is cool in many ways but a major pain to use for
anything approaching complex. The only reason we get away with it
in PDE build etc is because we autogen the build.xml files and effectively
unroll the loops and conditionals that one would normally write.
Jeff
Christof Marti <christof_marti@xxxxxxxxxx>
Sent by: platform-releng-dev-admin@xxxxxxxxxxx
09/10/2004 03:21 AM
|
Please respond to
platform-releng-dev |
|
|
To
| platform-releng-dev@xxxxxxxxxxx
|
|
cc
|
|
|
Subject
| Re: [platform-releng-dev]
again on session tests |
|
- the PDE/JUnit integration could be obtained
by using RepeatedTest from JUnit when run from a launch config
- you want to measure exactly the same thing
on each run _for the same scenario_, otherwise its another scenario (my
suggestion was for a single scenario, didn't make this explicit enough,
I think)
- the code snippet for repeating the same
scenario is (from PerformanceMeter's Javadoc):
public void testOpenEditor() {
Performance perf= Performance.getDefault();
PerformanceMeter performanceMeter= perf.createPerformanceMeter(perf.getDefaultScenarioId(this));
try {
for (int i= 0; i < 10; i++) {
performanceMeter.start();
operationToMeasure();
performanceMeter.stop();
}
performanceMeter.commit();
perf.assertPerformance(performanceMeter);
} finally {
performanceMeter.dispose();
}
}
I agree that ideally the same code would
work for session tests. To start a second instance just looks a bit heavyweight
to me. Ant must support loops, doesn't it?
Christof
Friday, September 10, 2004 12:35 AM
To: platform-releng-dev@xxxxxxxxxxx
cc:
From: Jeff McAffer <Jeff_McAffer@xxxxxxxxxx>
Subject: Re: [platform-releng-dev] again on session tests
Some points:
- scripting with Ant is really not much fun. It is both complex (see
John's message) and brittle (have to modify test.xml files everytime you
add a test etc)
- managing this through such scripts means that user cannot run the tests
easily from their workspace using the JUnit UI.
- the proposed "a way to" mechanisms are very specific to this
situation. What if I want to do other things differnetly on the last
run? or things differently on the first run as well?
The point here is that Rafael's approach has session tests be as much like
normal tests as possible. That allows users to code them naturally
and take advantage of all the other mechanisms available. WRT the
example below, assuming the tests were NOT session tests, what is the code
pattern for running multiple iterations of some test and asserting only
on the aggregate result? I think a design goal here should be to
make that same pattern work for session tests.
Jeff
p.s., I called them "patterns" just for Erich...
| Christof Marti/Zurich/IBM@IBMCH
09/09/2004 01:05 PM
|
|
To
| Rafael Chaves/Ottawa/IBM@IBMCA@IBMDE
|
|
cc
| Erich Gamma, Andre Weinand,
Jeff McAffer/Ottawa/IBM@IBMCA, Nick Edgar/Ottawa/IBM@IBMCA@IBMDE
|
|
Subject
| Re: [platform-releng-dev]
again on session tests |
|
For repeated measurements we suggest a solution where the test is run from
test.xml n times. Each run would tag its single measurement with some tag
'foo' and commit it to the database. Only the last run's assertPerformance()
would take all measurements with that tag 'foo' and compare them against
reference measurements.
What is need from the performance plug-in is:
- a way to tag runs with 'foo' from test.xml in all n runs
- a way to disable assertPerformance() from test.xml in the first n-1 runs
- a way to compare measurements taged with 'foo' against a reference in
the n-th run (like: assertPerformance(tag), where the tag would be 'foo'
in this example)
This approach would avoid the second instance of Eclipse and the additional
communication support from the second instance back to the first.
Christof
Thursday, September 09, 2004 5:48 PM
To: Christof Marti/Zurich/IBM@IBMCH
cc:
From: Rafael Chaves/Ottawa/IBM@IBMCA@IBMDE
Subject: Re: [platform-releng-dev] again on session tests
I am forwarding this directly to you because it seems we (Ottawa) are not
having our Notes messages being sent to non-Notes destinations (as the
platform-releng list).
Forwarding...
September 9, 2004 11:14 AM
To: platform-releng-dev@xxxxxxxxxxx
cc:
From: Rafael Chaves/Ottawa/IBM@IBMCA
Subject: Re: [platform-releng-dev] again on session tests
We have considered doing it the way you suggested and the reasons we didn't
go for it were:
a) flexibility/easy-of-use: developers would have to tweak their ant scripts
to create a separate target for every performance test. Integration with
PDE/Junit would not work (for most cases).
b) doing the measuring several times before commiting/asserting would not
be possible.
Rafael
Christof Marti <christof_marti@xxxxxxxxxx>
Christof Marti <christof_marti@xxxxxxxxxx>
Sent by: platform-releng-dev-admin@xxxxxxxxxxx
09/09/2004 06:21 AM
Please respond to
platform-releng-dev
To
platform-releng-dev@xxxxxxxxxxx
cc
Subject
Re: [platform-releng-dev] again on session tests
What is the reason for using a second instance of eclipse? Is the overhead
from the automated testing framework too big when registering a session
test directly in the test.xml?
Christof
Wednesday, September 08, 2004 10:55 PM
To: platform-releng-dev@xxxxxxxxxxx
cc:
From: Nick Edgar <Nick_Edgar@xxxxxxxxxx>
Subject: Re: [platform-releng-dev] again on session tests
Note that the Workbench has a very simple handshaking protocol for running
the tests once the workbench is ready. There is currently no support
for
inter-process communication though.
See ITestHarness and TestableObject. The design goals behind this
(small)
API were to provide maximum decoupling between the test harness, the
workbench, and the RCP app. For example, the RCP app doesn't need
to be
aware of the test harness at all, and the workbench does not need to know
about the PDE test runner. Ideally the PDE test runner wouldn't have
to
know about the workbench either, but it currently needs to obtain the
TestableObject from PlatformUI, so there's a dependency there.
Although these types currently exist at workbench level, they have no
dependencies on other workbench types. This was done in the hopes
that
they could eventually be generalized and pushed down. Maybe now
is the
time.
The control flow is:
- eclipse runs the test harness app (e.g. the PDE test runner), telling
it
which main app to run (e.g. the IDE)
- the test harness sets itself as the ITestHarness implementation on the
workbench's TestableObject:
PlatformUI.getTestableObject().setTestHarness(this)
- the test harness runs the main app (an RCP app, of course)
- the RCP app creates and runs the workbench as usual
(PlatformUI.createAndRunWorkbench(Display))
- the workbench initializes itself, restores any previously saved state,
etc.
- just before the workbench is about to run the event loop, it notifies
the test harness that it can run the tests (ITestHarness.runTests())
- this is actually done in a forked thread
- the test harness tells the TestableObject that it is about to
start
testing (TestableObject.testingStarting())
- this allows the workbench to put itself into test mode, e.g. make
sure
error dialogs don't pop up and hang the tests
- the test harness tells the TestableObject to run the tests
(TestableObject.runTest(Runnable))
- currently, due to difficulty in breaking up the flow of control
of
JUnit tests organized in a suite, all the tests are run in the same
runnable
- the workbench's TestableObject runs the tests in an asyncExec,
since
the UI tests expect to be run in the UI thread
- since runTests() was called from a forked thread, the workbench
fell
through to processing UI events, so it will process the asyncExec
when
there are no more OS events
- the test harness tells the TestableObject that it is done testing
(TestableObject.testingFinished())
- the workbench reverts back to non-test mode
Does this in any way map onto the kind of inter-process communication
you're doing now?
Thoughts?
Nick
Rafael Chaves/Ottawa/IBM@IBMCA
Sent by: platform-releng-dev-admin@xxxxxxxxxxx
09/08/2004 11:33 AM
Please respond to
platform-releng-dev
To
platform-releng-dev@xxxxxxxxxxx
cc
Subject
[platform-releng-dev] again on session tests
Platform/Core has been doing some work on running JUnit tests that end
up
spawning a second instance of Eclipse, or "session tests". There
is a lot
of scenarios that can only be tested with session tests, as startup
performance tests.
We have been doing some work in that area, and were curious if others had
the same need, and what approach was (or is planned to be) used to support
that.
Here is the approach we are taking:
- session tests use a special SessionTestSuite class to build their test
suites
- session tests are actually just proxies for the actual tests that know
which plug-in/class/method implements the test case
- when a session test is run, it launches a second instance of Eclipse
to
run a application that is/has a test runner. This test runner will run
that
single test case and report back the test result to the original instance
using a socket (right now we use org.eclipse.pde.junit.runtime test
applications and protocol for doing that, but it causes an undesirable
dependency on a non-platform plug-in). Once the test case has finished,
this second Eclipse instance terminates
- back to the original instance, we check whether the test failed/passed.
If it failed, we fake a corresponding error/failure to occur at the proxy
test, so it is reported as a regular test error/failure to the JUnit
framework
This way, session tests don't need any special support other than using
the
SessionTestSuite class when building their test suites. Also, the current
support for running automated tests as part of the RelEng test framework
or
by using the PDE/JDT support in the SDK still works.
Any feedback is welcome.
Rafael
_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev
_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev
_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev
