[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [platform-releng-dev] again on session tests
|
- the PDE/JUnit integration could be obtained by using RepeatedTest from
JUnit when run from a launch config
- you want to measure exactly the same thing on each run _for the same
scenario_, otherwise its another scenario (my suggestion was for a single
scenario, didn't make this explicit enough, I think)
- the code snippet for repeating the same scenario is (from
PerformanceMeter's Javadoc):
public void testOpenEditor() {
Performance perf=
Performance.getDefault();
PerformanceMeter
performanceMeter=
perf.createPerformanceMeter(perf.getDefaultScenarioId(this));
try
{
for (int i= 0; i < 10; i++)
{
performanceMeter.start();
operationToMeasure();
performanceMeter.stop();
}
performanceMeter.commit();
perf.assertPerformance(performanceMeter);
}
finally
{
performanceMeter.dispose();
}
}
I agree that ideally the same code would work for session tests. To start a
second instance just looks a bit heavyweight to me. Ant must support loops,
doesn't it?
Christof
Friday, September 10, 2004 12:35 AM
To:
platform-releng-dev@xxxxxxxxxxx
cc:
From: Jeff McAffer
<Jeff_McAffer@xxxxxxxxxx>
Subject: Re: [platform-releng-dev] again on
session tests
Some
points: - scripting with Ant is really
not much fun. It is both complex (see John's message) and brittle (have to
modify test.xml files everytime you add a test etc) - managing this through such scripts means that user
cannot run the tests easily from their workspace using the JUnit UI.
- the proposed "a way to" mechanisms are very
specific to this situation. What if I want to do other things differnetly
on the last run? or things differently on the first run as well?
The point here is that Rafael's approach
has session tests be as much like normal tests as possible. That allows
users to code them naturally and take advantage of all the other mechanisms
available. WRT the example below, assuming the tests were NOT session
tests, what is the code pattern for running multiple iterations of some test and
asserting only on the aggregate result? I think a design goal here should
be to make that same pattern work for session tests. Jeff p.s., I
called them "patterns" just for Erich...
| Christof
Marti/Zurich/IBM@IBMCH
09/09/2004 01:05 PM
|
|
To
| Rafael
Chaves/Ottawa/IBM@IBMCA@IBMDE
|
|
cc
| Erich Gamma, Andre
Weinand, Jeff McAffer/Ottawa/IBM@IBMCA, Nick
Edgar/Ottawa/IBM@IBMCA@IBMDE
|
|
Subject
| Re:
[platform-releng-dev] again on session
tests |
|
For repeated measurements we suggest a solution where the test is run
from test.xml n times. Each run would tag its single measurement with some tag
'foo' and commit it to the database. Only the last run's assertPerformance()
would take all measurements with that tag 'foo' and compare them against
reference measurements. What is need
from the performance plug-in is: - a way to
tag runs with 'foo' from test.xml in all n runs - a way to disable assertPerformance() from test.xml in the first n-1
runs - a way to compare measurements taged
with 'foo' against a reference in the n-th run (like: assertPerformance(tag),
where the tag would be 'foo' in this example) This approach would avoid the second instance of Eclipse and the
additional communication support from the second instance back to the
first. Christof Thursday, September 09, 2004 5:48 PM To: Christof Marti/Zurich/IBM@IBMCH cc: From:
Rafael Chaves/Ottawa/IBM@IBMCA@IBMDE Subject: Re: [platform-releng-dev] again on session tests
I am forwarding this
directly to you because it seems we (Ottawa) are not having our Notes messages
being sent to non-Notes destinations (as the platform-releng list).
Forwarding... September 9, 2004 11:14 AM To:
platform-releng-dev@xxxxxxxxxxx cc: From:
Rafael Chaves/Ottawa/IBM@IBMCA Subject: Re: [platform-releng-dev] again on session tests
We have considered doing it the way you
suggested and the reasons we didn't go for it were: a) flexibility/easy-of-use: developers would have to
tweak their ant scripts to create a separate target for every performance test.
Integration with PDE/Junit would not work (for most cases). b) doing the measuring several times before
commiting/asserting would not be possible. Rafael Christof Marti
<christof_marti@xxxxxxxxxx> Christof Marti <christof_marti@xxxxxxxxxx>
Sent by:
platform-releng-dev-admin@xxxxxxxxxxx 09/09/2004 06:21 AM
Please respond to
platform-releng-dev
To
platform-releng-dev@xxxxxxxxxxx
cc
Subject
Re: [platform-releng-dev] again on session tests
What is the reason for using a second instance of eclipse? Is the
overhead from the automated testing
framework too big when registering a session test directly in the test.xml? Christof Wednesday,
September 08, 2004 10:55 PM To:
platform-releng-dev@xxxxxxxxxxx cc: From: Nick Edgar
<Nick_Edgar@xxxxxxxxxx> Subject: Re: [platform-releng-dev] again on session tests
Note that the Workbench has a
very simple handshaking protocol for running the tests once the workbench is ready. There is currently no
support for inter-process
communication though. See
ITestHarness and TestableObject. The design goals behind this
(small) API were to provide maximum
decoupling between the test harness, the workbench, and the RCP app. For example, the RCP app doesn't need
to be aware of the test harness at
all, and the workbench does not need to know about the PDE test runner. Ideally the PDE test runner wouldn't
have to know about the workbench
either, but it currently needs to obtain the TestableObject from PlatformUI, so there's a dependency there.
Although these types currently exist at
workbench level, they have no dependencies on other workbench types. This was done in the hopes
that they could eventually be
generalized and pushed down. Maybe now is the time. The control flow is: - eclipse
runs the test harness app (e.g. the PDE test runner), telling it
which main app to run (e.g. the IDE)
- the test harness sets itself as the
ITestHarness implementation on the workbench's TestableObject: PlatformUI.getTestableObject().setTestHarness(this) - the test harness runs the main app (an RCP app, of
course) - the RCP app creates and
runs the workbench as usual (PlatformUI.createAndRunWorkbench(Display)) - the workbench initializes itself, restores any
previously saved state, etc.
- just before the workbench is about to run
the event loop, it notifies the test
harness that it can run the tests (ITestHarness.runTests()) - this is actually done in a forked
thread - the test harness
tells the TestableObject that it is about to start testing
(TestableObject.testingStarting()) - this allows the workbench to put itself into test mode, e.g.
make sure error dialogs don't
pop up and hang the tests -
the test harness tells the TestableObject to run the tests (TestableObject.runTest(Runnable))
- currently, due to difficulty in
breaking up the flow of control of JUnit tests organized in a suite, all the tests are run in the
same runnable - the workbench's TestableObject runs the tests
in an asyncExec, since the UI
tests expect to be run in the UI thread - since runTests() was called from a forked thread, the workbench
fell through to processing UI
events, so it will process the asyncExec when there are no more OS events - the test harness tells the TestableObject
that it is done testing
(TestableObject.testingFinished()) - the workbench reverts back to non-test mode Does this in any way map onto the kind of
inter-process communication you're
doing now? Thoughts?
Nick Rafael Chaves/Ottawa/IBM@IBMCA Sent by: platform-releng-dev-admin@xxxxxxxxxxx
09/08/2004 11:33 AM Please respond to platform-releng-dev To platform-releng-dev@xxxxxxxxxxx cc Subject
[platform-releng-dev] again on session
tests Platform/Core has been doing some work on running
JUnit tests that end up spawning a
second instance of Eclipse, or "session tests". There is a lot of scenarios that can only be tested with session
tests, as startup performance
tests. We have been doing some
work in that area, and were curious if others had the same need, and what approach was (or is planned to
be) used to support that.
Here is the approach we are
taking: - session tests use a special
SessionTestSuite class to build their test suites - session tests are
actually just proxies for the actual tests that know which plug-in/class/method implements the test
case - when a session test is run, it
launches a second instance of Eclipse to run a application that is/has a test runner. This test runner will
run that single test case and report back the test result to
the original instance using a socket
(right now we use org.eclipse.pde.junit.runtime test applications and protocol for doing that, but it
causes an undesirable dependency on a
non-platform plug-in). Once the test case has finished, this second Eclipse instance terminates
- back to the original instance, we check
whether the test failed/passed. If it
failed, we fake a corresponding error/failure to occur at the proxy
test, so it is reported as a regular test
error/failure to the JUnit framework This way,
session tests don't need any special support other than using the SessionTestSuite class when building their test suites. Also, the
current support for running automated
tests as part of the RelEng test framework or by using the PDE/JDT
support in the SDK still works. Any feedback is welcome. Rafael _______________________________________________ platform-releng-dev mailing list platform-releng-dev@xxxxxxxxxxx http://dev.eclipse.org/mailman/listinfo/platform-releng-dev
_______________________________________________ platform-releng-dev mailing list platform-releng-dev@xxxxxxxxxxx http://dev.eclipse.org/mailman/listinfo/platform-releng-dev
_______________________________________________ platform-releng-dev mailing list platform-releng-dev@xxxxxxxxxxx http://dev.eclipse.org/mailman/listinfo/platform-releng-dev
