Re: [platform-releng-dev] performance tests launching a 2nd instance of

[Christof Marti <christof_marti@xxxxxxxxxx>]

A simple solution could be to run a plugin test that calls snapshot(), commit(), assertPerformance(), dispose() on a meter (where snapshot() currently does not exist).

 

Christof

 

 

Wednesday, September 01, 2004 3:51 PM
To: platform-releng-dev@xxxxxxxxxxx
cc:
From: Jeff McAffer <Jeff_McAffer@xxxxxxxxxx>
Subject: Re: [platform-releng-dev] performance tests launching a 2nd instance of Eclipse

I think the main 
issue here is not how to deal with start/stop but rather how to deal with the 
results.  For most (all?) of the measurements we are interested in, the 
start values are 0 so there is no logical need to call start().  Allowing 
one to call stop() without a start() is error prone and poor form.  Rather, 
we should have a snapshot() or similar method which is roughly equivalent to 
calling stop() assuming that start() was called at time = 0.  The data is 
then managed and commited as normal (e.g., using commit() etc) and the results 
appear as a normal test.  There may be a further restriction that 
snapshot() cannot be called within a start()/stop() but that may not be 
needed. 

So, the main challenge is 
that this is all going on in a different VM.  So the test "target" itself 
can gather the data and commit it but who/how would do the assertions or other 
test pass/fail evaluation?  This is normally going on in the test 
"controller".  Some options: 

1) 
don't do assertions for these tests.  Simply assume that the commit will 
put the data in a file or a database and users can investigate the data through 
various reporting tools 

2) the 
commit() on the target writes the data somewhere (disk or database).  The 
test controller would then, in effect, load that data as a PerformanceMeter 
instance and do assertions etc. as if the test was actually run in 
process. 

3) others? 

Jeff

Christof Marti <christof_marti@xxxxxxxxxx>
Sent by: platform-releng-dev-admin@xxxxxxxxxxx

09/01/2004 06:24 AM

Please respond to
platform-releng-dev

To	platform-releng-dev@xxxxxxxxxxx
cc
Subject	Re: [platform-releng-dev] performance tests launching a 2nd instance of Eclipse

We currently do not have a solution for measuring 
startup time. A
suggestion that does not involve starting a second instance 
would be to
'somehow' invoke meter.start() right at the start (plugin 
activation of
o.e.test.performance is probably a problem?) and then 
meter.stop(),
.commit(), etc. in the first test that runs. A further 
suggestion would be
to, instead of invoking this as a usual plugin test from 
test.xml, call
start() in a special Java application that then directly 
invokes the
startup (within the same VM) of a plugin test like in the first 
suggestion.

Is this a possible way to go? Is there additional support you 
would need
from the performance plugin?

Christof


Wednesday, 
September 01, 2004 12:20 AM
To: 
platform-releng-dev@xxxxxxxxxxx
cc:
From: Rafael Chaves 
<Rafael_Chaves@xxxxxxxxxx>
Subject: [platform-releng-dev] performance 
tests launching a 2nd instance
of 
Eclipse







Platform/Core is working on startup 
performance tests. This requires being
able to launch a second instance of 
Eclipse whose startup time will be
measured (session test).

We could 
imagine two different approaches for that: the test case launches
a second 
instance, some code running in this second instance does the
measurements, 
and somehow communicates the results back to the test case,
running in the 
first instance. The test case would then make assertions on
the resulting 
data. Another would be to do all the measuring and assertions
in this 2nd 
instance, the test case in the first instance being only
responsible for 
spawning the second instance.

What is the vision for session tests? 
 I know the performance test
framework is an ongoing work, but how could 
one achieve that now (using
internal API if 
needed)?

Thanks,

Rafael

_______________________________________________
platform-releng-dev 
mailing 
list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev

_______________________________________________
platform-releng-dev 
mailing 
list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev