[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [platform-releng-dev] again on session tests
|
Let's not get stuck on philosophy... we really need these tests <g>
But... when I apply this philosophy to your scenario then I would
run the steps 1-7 inside a single JUnit test and make an assert at end.
For performance tests I would run 1-7 multiple times.
From your message below I conclude that you are exactly doing this,
so it looks like there is a match.
Some questions:
* such a test case doesn't have to run inside Eclipse. It basically
launches eclipse. Couldn't the "first Eclipse" therefore be
a vanilla JUnit test run which executes a vanilla JUnit test case?
* Since we use a database to collect the data you can actually
collect this from any process. Instead of transporting the
data back via a socket to the "first Eclipse" could you capture
directly from the "second Eclipse" ?
Regarding the ant complexity, I wonder whether some custom ant tasks
could simplify things, but I'm speculating.
--erich
John Arthorne
<John_Arthorne@ca
.ibm.com> To
Sent by: platform-releng-dev@xxxxxxxxxxx
platform-releng-d cc
ev-admin@eclipse.
org Subject
Re: [platform-releng-dev] again on
session tests
09/09/2004 08:00
PM
Please respond to
platform-releng-d
ev@xxxxxxxxxxx
This philosophy doesn't match the goal of the existing platform core
session tests. Their purpose is to assert that the workspace shutdown and
startup code is working properly. A typical scenario would be:
1) Start Eclipse
2) Create a project with a builder and some files
3) Run the builder
4) Modify some files in the workspace
5) Shutdown Eclipse
6) Restart Eclipse
7) Run the builder again and assert that the builder obtained a delta
matching the changes made in step 4)
Essentially, each session test is a series of steps (one step per method),
each of which requires a shutdown and restart of the *same* workspace. They
also test crash recovery, so sometimes we want a normal save and shutdown,
and other times we just want the VM to exit to simulate a crash. We have
tried to make this work with a single Eclipse instance driven by multiple
test.xml entries, but you end up with a very large number of test.xml
entries, with extra entries in between to cleanup the workspace state. We
found it easier in the end to have Eclipse within Eclipse tests, where the
first Eclipse has a simple application that does the setup/teardown for
each group of session tests.
Erich Gamma
<erich_gamma@xxxxxxxxxx>
Sent by: To
platform-releng-dev-admin@ec platform-releng-dev@xxxxxxxxxxx
lipse.org cc
platform-releng-dev@xxxxxxxxxxx
,
09/09/2004 11:49 AM platform-releng-dev-admin@eclip
se.org
Subject
Please respond to Re: [platform-releng-dev] again
platform-releng-dev on session tests
>+ *may* be natural in the true Junit philosophy (I confess to not fully
getting it)
The JUnit philosophy is that tests should leave the world exactly how they
found it.
This is support is built-in into JUnit and is explained (better than I
could
ever do it) in:
http://martinfowler.com/bliki/JunitNewInstance.html
Being that pure is not always practical. In particular when you
have to start a workbench for a test. So what we do is to run more than one
test in a single workspace. However, in this case it is your responsibility
to clean-up properly. One way to do it is explained in chapter 12 of
contributing
to eclipse:
http://today.java.net/today/2004/02/02/ch12Eclipse.pdf
>Don't we also have something like this to run the test suites in releng
mode?
org.eclipse.test releng includes corresponding applications which use an
Ant aware
TestRunner (EclipseTestRunner).
--erich
Jeff McAffer
<Jeff_McAffer@ca.
ibm.com> To
Sent by: platform-releng-dev@xxxxxxxxxxx
platform-releng-d cc
ev-admin@eclipse.
org Subject
Re: [platform-releng-dev] again on
session tests
09/09/2004 04:11
AM
Please respond to
platform-releng-d
ev@xxxxxxxxxxx
Mechanically this seems fine. It is basically using the mechanisms that we
have already. There is a question as to how people want to organize tests.
Using the support outlined here, each method in a TestCase class will spawn
a different VM. This has good (+) and bad (-) points.
+ simple
+ *may* be natural in the true Junit philosophy (I confess to not fully
getting it)
- makes writing/composing complex test "scenarios" harder as you do not get
to use the TestSuite and TestCase grouping/ordering mechanisms (e.g., if
you write a startup test, a CVS check out test and a full build test, you
cannot somehow then group these into a suite without forcing multiple VMs).
- may make it hard to talk about "scenarios" as a scenario would then just
map onto a particuarl method on a TestCase class.
As for the mechanics, depending on PDE is a bummer for this. We should
seek to push that function down or duplicate. Don't we also have something
like this to run the test suites in releng mode?
Jeff
Rafael Chaves/Ottawa/IBM@IBMCA
Sent by:
platform-releng-dev-admin@eclipse.o To
rg platform-releng-dev@ecli
pse.org
cc
09/08/2004 11:33 AM
Subject
[platform-releng-dev]
Please respond to again on session tests
platform-releng-dev
Platform/Core has been doing some work on running JUnit tests that end up
spawning a second instance of Eclipse, or "session tests". There is a lot
of scenarios that can only be tested with session tests, as startup
performance tests.
We have been doing some work in that area, and were curious if others had
the same need, and what approach was (or is planned to be) used to support
that.
Here is the approach we are taking:
- session tests use a special SessionTestSuite class to build their test
suites
- session tests are actually just proxies for the actual tests that know
which plug-in/class/method implements the test case
- when a session test is run, it launches a second instance of Eclipse to
run a application that is/has a test runner. This test runner will run that
single test case and report back the test result to the original instance
using a socket (right now we use org.eclipse.pde.junit.runtime test
applications and protocol for doing that, but it causes an undesirable
dependency on a non-platform plug-in). Once the test case has finished,
this second Eclipse instance terminates
- back to the original instance, we check whether the test failed/passed.
If it failed, we fake a corresponding error/failure to occur at the proxy
test, so it is reported as a regular test error/failure to the JUnit
framework
This way, session tests don't need any special support other than using the
SessionTestSuite class when building their test suites. Also, the current
support for running automated tests as part of the RelEng test framework or
by using the PDE/JDT support in the SDK still works.
Any feedback is welcome.
Rafael
_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev
_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/platform-releng-dev
