[ecf-dev] Re: Automated test execution for ECF build based on autotestsu

["Ted Kubaska (easy)" <tkubaska@xxxxxxxxxxxxxx>]

Thanks Marcus.

I apologize for not making the ECF phone call ... I was traveling. I'm
responding just to let you know your mail hasn't gone into a black
hole. 

What you say makes a lot of sense. Please let me study it and get back
to you in a week (8/19). I'm also traveling for the next week, but
should have some limited email access.
 -ted




On Fri, 2008-08-08 at 10:31 +0200, Markus Alexander Kuppe wrote:
> Hi Ted,
> 
> automated testing was one of yesterday's topics the ECF con call. Since 
> I've integrated automated test execution into Versants build process 
> recently, I'd like to share how it's done. You have probably an idea 
> where it's best integrated into the ECF build.
> 
> I first looked into the Eclipse testing framework [1]. It's JUnit3 based 
> and spawns of an Eclipse instance for test execution. It supports UI 
> tests, headless tests as well as performance tests. Test execution is 
> handled by Ant. It spawns an Eclipse instance and executes test suites. 
> I find this way too complex and cumbersome. A developer simply wants to 
> write a unit test. One doesn't want write/maintain xml that defines 
> execution order. Especially since execution order shouldn't matter for 
> unit tests.
> 
> Thus I continued to look for something that doesn't require additional 
> xml next to the plain test cases and I found pluginbuilder autotestsuite 
> (EPL) [2]. It consists of a bunch of OSGi bundles which come with an 
> Equinox application. Basically one drops the unit tests + autotestsuite 
> bundles into the Eclipse installation under test and starts the 
> autotestsuite application (-application 
> org.pluginbuilder.autotestsuite.application.AutoTestApplication). 
> Autotestsuite then takes care of finding test cases by searching bundles 
> and bundle classpaths based on regular expression. All test cases that 
> are found are passed to a junit.framework.TestSuite and executed. Test 
> results are collected by the junit xml formatter. Autotestsuite doesn't 
> require lists of test cases nor hard coded junit.framework.TestSuites 
> and IMO that's the beauty of it.
> 
> So how could the integration of autotestsuite in ECF look like?
> 1) Build test bundles. I'd suggest to create test features and build and 
> package them independently. Like it is done for platform [3].
> 2) Deploy Eclipse SDK + ECF + ECF tests + autotestsuite to some temp 
> folder on the build box, either by unzipping everything into features/ 
> and plugins/ (<3.4), using the links/ folder (that's how Versant does 
> it) or by using the p2 director.
> 3) Launch the autotestsuite application.
> 4) Collect the JUnit xml output and post process it (html reports, email 
> notification...)
> 
> Cheers
> Markus
> 
> [1] 
> http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.test/testframework.html
> [2] http://www.pluginbuilder.org/documentation/run-tests/
> [3] 
> http://download.eclipse.org/eclipse/downloads/drops/R-3.4-200806172000/download.php?dropFile=eclipse-Automated-Tests-3.4.zip