Re: [platform-releng-dev] again on session tests

[John Arthorne <John_Arthorne@xxxxxxxxxx>]

This philosophy doesn't match the goal
of the existing platform core session tests. Their purpose is to assert
that the workspace shutdown and startup code is working properly.  A
typical scenario would be:

1) Start Eclipse
2) Create a project with a builder and
some files
3) Run the builder
4) Modify some files in the workspace
5) Shutdown Eclipse
6) Restart Eclipse
7) Run the builder again and assert
that the builder obtained a delta matching the changes made in step 4)

Essentially, each session test is a
series of steps (one step per method), each of which requires a shutdown
and restart of the *same* workspace. They also test crash recovery, so
sometimes we want a normal save and shutdown, and other times we just want
the VM to exit to simulate a crash. We have tried to make this work with
a single Eclipse instance driven by multiple test.xml entries, but you
end up with a very large number of test.xml entries, with extra entries
in between to cleanup the workspace state. We found it easier in the end
to have Eclipse within Eclipse tests, where the first Eclipse has a simple
application that does the setup/teardown for each group of session tests.

Erich Gamma <erich_gamma@xxxxxxxxxx>
Sent by: platform-releng-dev-admin@xxxxxxxxxxx

09/09/2004 11:49 AM

Please respond to
platform-releng-dev

To	platform-releng-dev@xxxxxxxxxxx
cc	platform-releng-dev@xxxxxxxxxxx, platform-releng-dev-admin@xxxxxxxxxxx
Subject	Re: [platform-releng-dev] again on session tests

>+ *may* be natural in the true Junit philosophy (I confess to not fully
getting it)
The JUnit philosophy is that tests should leave the world exactly how they
found it.
This is support is built-in into JUnit and is explained (better than I
could
ever do it) in:
      http://martinfowler.com/bliki/JunitNewInstance.html

Being that pure is not always practical. In particular when you
have to start a workbench for a test. So what we do is to run more than
one
test in a single workspace. However, in this case it is your responsibility
to clean-up properly. One way to do it is explained in chapter 12 of
contributing
to eclipse:
      http://today.java.net/today/2004/02/02/ch12Eclipse.pdf

>Don't we also have something like this to run the test suites in releng
mode?
org.eclipse.test releng includes corresponding applications which use an
Ant aware
TestRunner (EclipseTestRunner).

--erich



                    
                     
                     
          
             Jeff McAffer      
                     
                     
             <Jeff_McAffer@ca.  
                     
                    
             ibm.com>      
                     
                     
To 
             Sent by:      
           platform-releng-dev@xxxxxxxxxxx
    
             platform-releng-d    
                     
               cc 
             ev-admin@eclipse.    
                     
                  
             org        
                     
                    Subject
                    
                  Re: [platform-releng-dev]
again on  
                    
                  session
tests                    
  
             09/09/2004 04:11    
                     
                   
             AM        
                     
                     
       
                    
                     
                     
          
                    
                     
                     
          
             Please respond to    
                     
                  
             platform-releng-d    
                     
                  
              ev@xxxxxxxxxxx  
                     
                     
                    
                     
                     
          
                    
                     
                     
          





Mechanically this seems fine. It is basically using the mechanisms that
we
have already.  There is a question as to how people want to organize
tests.
Using the support outlined here, each method in a TestCase class will spawn
a different VM.  This has good (+) and bad (-) points.
+ simple
+ *may* be natural in the true Junit philosophy (I confess to not fully
getting it)
- makes writing/composing complex test "scenarios" harder as
you do not get
to use the TestSuite and TestCase grouping/ordering mechanisms (e.g., if
you write a startup test, a CVS check out test and a full build test, you
cannot somehow then group these into a suite without forcing multiple VMs).

- may make it hard to talk about "scenarios" as a scenario would
then just
map onto a particuarl method on a TestCase class.

As for the mechanics, depending on PDE is a bummer for this.  We should
seek to push that function down or duplicate.  Don't we also have
something
like this to run the test suites in releng mode?

Jeff


                    
                     
                     
          
 Rafael Chaves/Ottawa/IBM@IBMCA            
                     
         
 Sent by:                  
                     
                     
   
 platform-releng-dev-admin@eclipse.o          
                     
   To 
 rg                    
                     
    platform-releng-dev@ecli 
                    
                     
       pse.org            
     
                    
                     
                     
       cc 
 09/08/2004 11:33 AM                
                     
                
                    
                     
                     
  Subject 
                    
                     
       [platform-releng-dev]    
          Please respond to      
                again on session
tests   
         platform-releng-dev        
                     
                
                    
                     
                     
          
                    
                     
                     
          
                    
                     
                     
          
                    
                     
                     
          
                    
                     
                     
          
                    
                     
                     
          









Platform/Core has been doing some work on running JUnit tests that end
up
spawning a second instance of Eclipse, or "session tests". There
is a lot
of scenarios that can only be tested with session tests, as startup
performance tests.

We have been doing some work in that area, and were curious if others had
the same need, and what approach was (or is planned to be) used to support
that.

Here is the approach we are taking:
- session tests use a special SessionTestSuite class to build their test
suites
- session tests are actually just proxies for the actual tests that know
which plug-in/class/method implements the test case
- when a session test is run, it launches a second instance of Eclipse
to
run a application that is/has a test runner. This test runner will run
that
single test case and report back the test result to the original instance
using a socket (right now we use org.eclipse.pde.junit.runtime test
applications and protocol for doing that, but it causes an undesirable
dependency on a non-platform plug-in). Once the test case has finished,
this second Eclipse instance terminates
- back to the original instance, we check whether the test failed/passed.
If it failed, we fake a corresponding error/failure to occur at the proxy
test, so it is reported as a regular test error/failure to the JUnit
framework

This way, session tests don't need any special support other than using
the
SessionTestSuite class when building their test suites. Also, the current
support for running automated tests as part of the RelEng test framework
or
by using the PDE/JDT support in the SDK still works.

Any feedback is welcome.

Rafael

_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev


_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev