Re: [platform-releng-dev] performance tests launching a 2nd instance of

[Andre Weinand <eclipse@xxxxxxxxxxx>]

Jeff,

Yes, a "snapshot()" seems more intuitive than 
"stopMeasuring()" and we are planning to
provide something like that in the future.
But for now we are reluctant to add additional 
API before all consequences are understood.

But this shouldn't block you from implementing your use cases.

Both are easily solved by using two PerformanceMeters:

1)
	Performance p= Performance.getDefault();
	PerformanceMeter pm1= p.createPerformanceMeter("Scenario 1");
	pm1.stop();

	<do more stuff>
	PerformanceMeter pm2= p.createPerformanceMeter("Scenario 2");
	pm2.stop();

	// save data to DB
	pm1.commit();
	pm2.commit();

	p.assert(p1);
	p.assert(p2);

2)
	Performance p= Performance.getDefault();
	PerformanceMeter pm1= p.createPerformanceMeter("Scenario 1");
	PerformanceMeter pm2= p.createPerformanceMeter("Scenario 2");

        pm1.start();
        <do stuff>
        pm1.stop();
	pm1.commit();
        pm1.assert();

        pm2.start();
        <do more stuff>
        pm2.stop();
	pm2.commit();
        pm2.assert();

In the latter case it might be better to move 
both commits and asserts to the end so that the
measurements are not affected by Cloudscape's class loading.

Thanks for your feedback,
--andre



> Andre,
>
> I'm not sure of the semantics of stopMeasuring() 
> but it feels a little funny.  What we really 
> want to do here is "snapshot".  We may actually 
> wnat to continue measuring things and do 
> start()/stop() pairs.  We may even want to do 
> further "snapshots".  Without this we would 
> never be able to declare absolute bounds on 
> things.  For example, lets say we do a mess of 
> different things but at various points want to 
> assert that we are using no more than X memory 
> and Y handles (whatever).  This should be 
> something like
>
>         <do stuff>
>         snapshot();
>         assert();
>         <do more stuff>
>         snapshot();
>         assert();
>         ...
>
> Just as one might do
>         start();
>         <do stuff>
>         stop();
>         assert();
>         start();
>         <do more stuff>
>         stop();
>         assert();
>
> In this model, snapshot() gets described as 
> equivalent to stop() with an implict start() at 
> time = 0.
>
> Jeff
>
> Andre Weinand <eclipse@xxxxxxxxxxx>
> Sent by: platform-releng-dev-admin@xxxxxxxxxxx
>
> 09/06/2004 03:30 PM
> Please respond to
> platform-releng-dev
>
> To
> platform-releng-dev@xxxxxxxxxxx
> cc
> Rafael Chaves/Ottawa/IBM@IBMCA
> Subject
> Re: [platform-releng-dev] performance tests 
> launching a 2nd instance of Eclipse
>
>
>
>
>
>
> Hi Rafael,
>
> In a second iteration I've added more API to
> org.eclipse.performance so that my (probably
> naïve) StartupTest reduces to:
>
>
> import org.eclipse.test.performance.*;
>
> public class StartupTest extends PerformanceTestCase {
>
>                 public void testStartup() throws Exception {
>                     stopMeasuring();
>                     commitMeasurements();
>
> assertPerformanceInRelativeBand(Dimension.ELAPSED_PROCESS,
> -50, +5);
>                 }
> }
>
>
> The new method assertPerformanceInRelativeBand
> (both on PerformanceTestCase as well as the
> Performance façade) lets you specify the
> dimension and the lower and upper bound of the
> band as a percentage.
>
> So in the example from above, the test will fail
> if the startup time takes 5% longer or becomes
> 50% faster than some reference value (the 50%
> lower bound was chosen because such a dramatic
> performance improvement most likely is an
> indication for some other problem and not a real
> improvement; however if such an improvement is in
> fact possible, just specify "-100, +5"  :-)
>
> Let me know whether this works for you.
> Thanks,
> --andre
>
>
>
>
> At 18:20 Uhr -0400 31.8.2004, Rafael Chaves wrote:
> > Platform/Core is working on startup performance tests. This requires being
> > able to launch a second instance of Eclipse whose startup time will be
> > measured (session test).
> >
> > We could imagine two different approaches for that: the test case launches
> > a second instance, some code running in this second instance does the
>  >measurements, and somehow communicates the results back to the test case,
> > running in the first instance. The test case would then make assertions on
> > the resulting data. Another would be to do all the measuring and assertions
> > in this 2nd instance, the test case in the first instance being only
> > responsible for spawning the second instance.
> >
> > What is the vision for session tests?  I know the performance test
> > framework is an ongoing work, but how could one achieve that now (using
> > internal API if needed)?
> >
> > Thanks,
> >
> > Rafael
> >
> > _______________________________________________
> > platform-releng-dev mailing list
> > platform-releng-dev@xxxxxxxxxxx
> > http://dev.eclipse.org/mailman/listinfo/platform-releng-dev
>
> _______________________________________________
> platform-releng-dev mailing list
> platform-releng-dev@xxxxxxxxxxx
> http://dev.eclipse.org/mailman/listinfo/platform-releng-dev