[platform-releng-dev] ANN: Performance test infrastructure

[Andre Weinand <eclipse@xxxxxxxxxxx>]

Hi,
we've released a first cut of the performance test infrastructure. It 
is based on the existing code from the org.eclipse.perfmsr and Text 
team's work.

The performance infrastructure lives in a single plugin (and a 
fragment for Windows):
             org.eclipse.test.performance
             org.eclipse.test.performance.win32

Its API is the result of experience gained from various performance 
tests developed by the text team. It is fairly thin and has only 
these three classes:
             org.eclipse.test.performance.Performance
             org.eclipse.test.performance.PerformanceMeter
             org.eclipse.test.performance.PerformanceTestCase

Performance is basically a factory for PerformenceMeters. You write a 
performance test by using a PerformanceMeter to collect timing (and 
other) data before and after the operation under test.
Multiple runs are supported as well. At the end of a test the 
collected data can be stored in a local or remote database either to 
define a reference data set or just to accumulate historical data. A 
call to assertPerformance assures that the collected data indicates 
no performance degradation.

An example usage in a test case looks like this:

   Performance perf= Performance.getDefault();
   PerformanceMeter perfMeter=
      perf.createPerformanceMeter(perf.getDefaultScenarioId(this));
   try {
      for (int i= 0; i < 10; i++) {
         performanceMeter.start();
         // operation under test
         performanceMeter.stop();
      }
      performanceMeter.commit();
      perf.assertPerformance(performanceMeter);
   } finally {
      performanceMeter.dispose();
   }

The class PerformanceTestCase is a convenience class that makes the 
use of the PerformanceMeter transparent.

With it the code from above becomes:

public class MyPerformanceTestCase extends PeformanceTestCase {

	public void testMyOperation() {
		for (int i= 0; i < 10; i++) {
			startMeasuring();
			// my operation
			stopMeasuring();
		}
		commitMeasurements();
		assertPerformance();
	}
}


Please note: since the current implementation does not store data in 
a database, the assertPerformance method is disabled (and won't 
fail). This limitation doesn't block you from starting to implement 
performance tests.

However, in order to see results, you can define the (temporary) 
property "InternalPrintPerformanceResults" as a VM argument in the 
launch config or pass it on the command line:
	-DInternalPrintPerformanceResults

When this property is set, PerformanceMeter.commit() will dump the 
collected data to the console.

There is currently no support to visualize the result of performance tests.
This and a more detailled "How to..." document will follow soon...

Any feedback is welcome
--andre