[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [epsilon-dev] Tests in Jenkins
|
Hi Sina,
Thanks for all the hard work on getting the tests to run outside
Eclipse! All ETL tests seem to be still passing when run within
Eclipse, although I've noticed some exceptions related to cleaning up
temp files/directories. Related to that, I'm wondering whether
Files.createTempDirectory/createTempFile could replace the respective
homegrown functions in FileUtil. Also, I wouldn't worry about cleaning
up temporary files and directories created using Files.* methods; I'd
leave it to the OS.
Thanks,
Dimitris
On Mon, 4 May 2020 at 00:14, Sina Madani <sinadoom@xxxxxxxxxxxxxx> wrote:
>
> Hi everyone,
>
>
>
> I’ve been working on trying to get the test suite working via Maven (following on from Horacio’s work on this) on the smadani/unittests branch and I’ve had some success in fixing some of the issues, though not everything is working flawlessly. As a reminder, the main difference is that we write all resources to a temporary directory so that we’re not relying on having the src folder available, meaning tests can be run from outside of Eclipse. Unfortunately the temp folder isn’t cleaned up properly and there area few failures but it’s an improvement on the state of things before.
>
>
>
> I’ve managed to get rid of Hamcrest altogether since in pretty much every case it was being used like assertEquals / assertTrue anyway and it caused more problems than it solved (deprecations, an unnecessary dependency to manage etc.), so that helped.
>
>
>
> The current setup in the branch is to do mvn surefire:test (as Horacio suggested) on the org.eclipse.epsilon.test plugin once everything’s built. As explained there’s some issues with HUTN but that’s about it as far as I can see on the Maven side of things.
>
>
>
> Due to the refactoring to creating temp files, there is only one annoying test that I’ve spent almost a full day trying to figure out why it won’t work and that’s Traceability in EGL (the Hierarchy one, though I suspect only Louis would know how all this traceability stuff in EGL works). There’s also this weird issue with ETL where it seems that the transformation is in-place so when testing with multiple IEtlModule implementations / parameters the oracle needs to be copied back over (no other reason I can think of for failures) and sometimes, only for the Tree2Graph test, sometimes it fails all the tests and sometimes it doesn’t, even though the exact same methodology is used as the OO2DB test. Definitely something dodgy going on there, perhaps with EMF resource registration or something.
>
>
>
> Even if we don’t fix these problems, that’s still the vast majority of the tests working, so I suppose it’s better than nothing. We could comment out these tests that aren’t working for the Maven build (i.e. create a separate suite) and only run them locally (i.e. from within Eclipse) as a workaround, which we need to do for some tests anyway (for example the Simulink tests won’t work on CI if Simulink isn’t set up). We could then re-use the old way of running the tests (i.e. relying on being able to navigate to the src folder using X.class.getResource(“”).toString().replace(“bin”, “src”) or whatever) for these troublesome tests rather than temp files.
>
>
>
>
>
> Thanks,
>
> Sina
>
>
>
> From: Antonio Garcia-Dominguez
> Sent: 14 February 2019 20:03
> To: Epislon Project developer discussions
> Subject: Re: [epsilon-dev] Tests in Jenkins
>
>
>
> Hi Horacio,
>
>
>
> Why do we have multiple hamcrest versions? Would it be possible to unify to a specific hamcrest version?
>
>
>
> Pure Maven won't care about the Eclipse metadata: the build will be based entirely on dependencies. Surefire can exclude dependencies if you have a problem with those:
>
>
>
> https://maven.apache.org/surefire/maven-surefire-plugin/examples/configuring-classpath.html
>
>
>
> Cheers,
>
> Antonio
>
>
>
> On Thu, 14 Feb 2019 at 15:00, Horacio Hoyos <arcanefoam@xxxxxxxxx> wrote:
>
> Hi all,
>
>
>
> I have been working on this problem and there are several issues and challenges associated with making this work. I’ll try to point them out individually, if I can untangle the mess in my head.
>
>
>
> 1. Tycho surefire plugin. It seems that Tycho takes over the tests, which is good to run the plug-in tests. However, running the unit tests from within a nested eclipse fails due to discrepancies between versions of the compiled classes.
>
> - A workaround is to split the tests plugin into two, and have a separate tests.unit plugin which we use to run the tests. For pros/cons see points:
>
>
>
> 2, Junit vs Maven. Making JUnit directly run the tests (i.e. mimicking what we do from Eclipse run configurations) is a PITA because it requires to setup the classpath. However, this has the advantage that the file access (for test models, metamodels and scripts) as currently done would work. Running the tests via maven we no longer have to worry about the class path. However, maven runs the tests from the compiled classes, which means that file access has to be changed to access resources within classes. In my proposed attempt I went with the mvn path (but has some side effects too).
>
>
>
> 3. The current file resolution approach woks as expected with scripts/models that import other models within JUnit because since we access files in the sources folder the paths to imported files work. When reading resources from classes this does not work.
>
>
>
> 4. Some HUTN tests expect the Family.ecore metramodel to be registered (I think), but it gets registered with the class resource path so then it can not be used… or something like that. Needs more investigation.
>
>
>
> My ideas:
>
>
>
> - As I said before I went via the maven approach and created a separate plugin for the unit tests.
>
>
>
> - The FileUitl class has been modified to make use of temp files to create temp versions of the required models and scripts; it also respects the folder hierarchy so relative paths work . I have changed a lot of the tests to play nicely with this change. But the results tell me that there are still a couple of places where some missing imported or references models are missing.
>
>
>
> - The tests can be invoked using mvn surefire:test. Explicit call to maven’s surefire needs to be done to avoid Tycho’s surefire. This command has to be executed after the main maven build has finished.
>
>
>
> - There are several test that fail atm. Some because of assertions, other for missing files and other for the missing Families meta model. Further, I am always getting out of memory errors when some of the new tests try to start separate threads. I am using Xmx6166M and still getting this errors. More investigation needs to be done to see if the failing tests are because of actual errors or because as a result of issues on creating the temp files. These errors needs further investigation because I think the results from the mvn run are different from the ones I get from running within eclipse.
>
>
>
> - One of the effects of the change is that some tests fail from within eclipse with a Security exception caused by conflicts between hamcrest versions. I have tried to solve this issue by changing the import order of dependencies, changing where dependencies are imported, etc. and can’t figure out what else to do. I am not happy that we would always need to run the tests from the command line.
>
>
>
> Any ideas or help on this is welcome. I have pushed this idea to hoyos/unittests
>
>
>
> Cheers,
>
>
>
>
>
> On 12 Jan 2019, at 16:16, Sina Madani <sinadoom@xxxxxxxxxxxxxx> wrote:
>
>
>
> Hi Dimitris,
>
>
>
> I think this is a great idea, and only discovered that EpsilonTestSuite was not run as part of mvn clean install recently. If we can guarantee that the tests will be run in a Unix environment then this would work. Currently there are a number of tests that fail on Windows and even under Linux due to the aforementioned file locations, permissions etc. Perhaps we could start with a smaller subset and gradually add the tests which are currently failing in certain circumstances once fixed? Also the Xmx needs to be adjusted to at least 4G (I think I may have already done this) to accommodate the full test suite.
>
>
>
> Thanks,
>
> Sina
>
> _______________________________________________
> epsilon-dev mailing list
> epsilon-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/epsilon-dev
>
>
>
> Arcanefoam
>
>
>
> https://bitbucket.org/arcanefoam/
>
> https://github.com/arcanefoam
>
>
>
>
>
>
>
> _______________________________________________
> epsilon-dev mailing list
> epsilon-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/epsilon-dev
>
>
>
>
> --
>
> Antonio Garcia-Dominguez
>
>
>
> _______________________________________________
> epsilon-dev mailing list
> epsilon-dev@xxxxxxxxxxx
> To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epsilon-dev
--
Dimitris Kolovos
Professor of Software Engineering
Department of Computer Science
University of York
http://www.cs.york.ac.uk/~dkolovos
EMAIL DISCLAIMER http://www.york.ac.uk/docs/disclaimer/email.htm
