Re: [incquery-dev] Jenkins build is still unstable: EMF-IncQuery Testing

[Ujhelyi Zoltán <ujhelyiz@xxxxxxxxxx>]

Hi,

these test failures were brough forth by a migrating the builds to a different server instance. After evaluating it in our Eclipse instance, we have found with Ábel that this issue comes only forth if all tests of the incquery.runtime.base.tests projects are executed, but works well if they are evaluated isolatedly.

After some experimenting with git bisect I have found an old commit (https://github.com/ujhelyiz/EMF-IncQuery-Examples/commit/997f93fe8411ec735b3a3216b947e7f75858a570) that introduces such an issue first: it changes the test environment to re-use the same model instance between different test cases.

This means, not the failing tests are faulty here but some other test that changes the models, but fails to restore its model. We will need to track that down, but I have to leave for now.

On the other hand, I am asking whether this approach was a good idea. My main problem with this kind of reuse is that a single faulty test can break other tests in a non-deterministic ways. The original approach that re-built the model for each test case is much more safe here.

Cheers,
Zoli
-- Zoltán Ujhelyi
https://www.inf.mit.bme.hu/en/members/ujhelyiz

Fault Tolerant Systems Research Group
Budapest University of Technology and Economics

On 2013.10.31., at 16:11, Jenkins Build  <build@xxxxxxxxxxxxxx> wrote:

> See <https://build.inf.mit.bme.hu/jenkins/job/EMF-IncQuery-Tests-School/changes>