Re: [virgo-dev] Test framework for MultiThreaded tests

[Glyn Normington <gnormington@xxxxxxxxxx>]

Hi Olivier

Interesting. Thanks for writing this up, and for your kind words about our code quality.

Note that if you want to use MultithreadedTC in Virgo we'll need to put it through the Ecliose legal process. This isn't a problem but will introduce a delay into the process until MultithreadedTC is approved. After that we'll be free to use it over and over.

Regards,
Glyn
On 13 Aug 2010, at 23:58, Olivier Girardot wrote:

Hi everyone,
i did a little bit of testings (...yes, again) but this time with MultiThreadedTC and did a blog post on it and my opinion.
To sum it up, i think this test framework can greatly help several points :
  * handlind errors
  * handling thread synchronization
and finally
  * improve tests "easiness"

You can check out the whole article here : http://bit.ly/cXEFHw
i present a comparison between two ways of handling a test case ( a real one),

Thank you for your time and 'night everyone.
Regards,

Olivier.

2010/8/11 Olivier Girardot <ssaboum@xxxxxxxxx<mailto:ssaboum@xxxxxxxxx>>
Hi everyone,
i edited the wiki page Steve P. created about multi-threaded tests, i did a lot of testing and research to find a clean and easy way to handle the problem we found out during the creation of the test case https://bugs.eclipse.org/bugs/show_bug.cgi?id=321397 . Steve summed it up in a clear manner on the wiki :


 1.  Child threads that make assertions have the results of those assertions undetected.
 2.  The (child) threads can terminate before or after the main thread (which is running the JUnit test), so even if the child assertions are correctly reported, there may be no-one to report them to. (Note that the overall result of a test needs to be gathered, too, so some sort of synchronisation is vital.)
 3.  Mock objects (which make implicit assertions during use and on verify) shared between threads can have their assertions caught (as in the answer to the first problem) but verify calls need to be made after all the child threads have finished, or at least not too soon.

So what should we do ?
I thought about many ways we could handle that but the perfect solution isn't designed yet, so i submitted the following proposals :

 *   (old one) Brian Goetz mentionned in Java Concurrency in Practice that for the JSR 166 (creating the java.util.concurrent package) a test framework was created to handle assertions error in generated threads. I found back the class : JSR166TestCase<http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/test/tck/JSR166TestCase.java?view=markup> and we should take a look;

 *   We can handle these problems using ThreadGroups exceptions handling, but exceptions will be ducked back into the "root" ThreadGroup. Killing potentially generated Threads;

 *   Study how the Junit extension [MultithreadedTestCase<http://code.google.com/p/multithreadedtc/>] handles it and how we can learn from it;

 *   (new one) Using aspects to watch for AssertionsError, no matter what thread will launch it. This is the less intrusive way but its scope must be carefully studied. (Binary weaving can be painfull (especially on the 16 000 jdk classes), i think we should use Load Time Weaving as much as possible but it has to be tested).

I'd love some reviews on these and know what you think

Regards,

Olivier Girardot.

<ATT00001..txt>