Re: [platform-releng-dev] performance tests launching a 2nd instance of

[Jeff McAffer <Jeff_McAffer@xxxxxxxxxx>]

Andre, 

I'm not sure of the semantics of stopMeasuring()
but it feels a little funny.  What we really want to do here is "snapshot".
 We may actually wnat to continue measuring things and do start()/stop()
pairs.  We may even want to do further "snapshots".  Without
this we would never be able to declare absolute bounds on things.  For
example, lets say we do a mess of different things but at various points
want to assert that we are using no more than X memory and Y handles (whatever).
 This should be something like 

        <do
stuff>
        snapshot();
        assert();
        <do
more stuff>
        snapshot();
        assert();
        ...

Just as one might do
        start();
        <do
stuff>
        stop();
        assert();
        start();
        <do
more stuff>
        stop();
        assert();

In this model, snapshot() gets described
as equivalent to stop() with an implict start() at time = 0.

Jeff

Andre Weinand <eclipse@xxxxxxxxxxx>
Sent by: platform-releng-dev-admin@xxxxxxxxxxx

09/06/2004 03:30 PM

Please respond to
platform-releng-dev

To	platform-releng-dev@xxxxxxxxxxx
cc	Rafael Chaves/Ottawa/IBM@IBMCA
Subject	Re: [platform-releng-dev] performance tests launching a 2nd instance of Eclipse

Hi Rafael,

In a second iteration I've added more API to 
org.eclipse.performance so that my (probably 
naïve) StartupTest reduces to:


import org.eclipse.test.performance.*;

public class StartupTest extends PerformanceTestCase {

                
public void testStartup() throws Exception {
                
    stopMeasuring();
                
    commitMeasurements();
 
assertPerformanceInRelativeBand(Dimension.ELAPSED_PROCESS, 
-50, +5);
                
}
}


The new method assertPerformanceInRelativeBand 
(both on PerformanceTestCase as well as the 
Performance façade) lets you specify the 
dimension and the lower and upper bound of the 
band as a percentage.

So in the example from above, the test will fail 
if the startup time takes 5% longer or becomes 
50% faster than some reference value (the 50% 
lower bound was chosen because such a dramatic 
performance improvement most likely is an 
indication for some other problem and not a real 
improvement; however if such an improvement is in 
fact possible, just specify "-100, +5"  :-)

Let me know whether this works for you.
Thanks,
--andre




At 18:20 Uhr -0400 31.8.2004, Rafael Chaves wrote:
>Platform/Core is working on startup performance tests. This requires
being
>able to launch a second instance of Eclipse whose startup time will
be
>measured (session test).
>
>We could imagine two different approaches for that: the test case launches
>a second instance, some code running in this second instance does the
>measurements, and somehow communicates the results back to the test
case,
>running in the first instance. The test case would then make assertions
on
>the resulting data. Another would be to do all the measuring and assertions
>in this 2nd instance, the test case in the first instance being only
>responsible for spawning the second instance.
>
>What is the vision for session tests?  I know the performance
test
>framework is an ongoing work, but how could one achieve that now (using
>internal API if needed)?
>
>Thanks,
>
>Rafael
>
>_______________________________________________
>platform-releng-dev mailing list
>platform-releng-dev@xxxxxxxxxxx
>http://dev.eclipse.org/mailman/listinfo/platform-releng-dev

_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev