Hi Christian,
How did you monitor the time spent by maven ? Do you have a report for each step in the lifecycle?
One solution to help maven would be refactor the top-pom-main to have real module management.
For example replace :
<modules>…
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.common.core</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.common.sdk.core</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.common.ui</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.custom.core</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.custom.metamodel</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.custom.metamodel.edit</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.custom.sdk.core</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.custom.sdk.ui</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.custom.ui</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.efacet</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.efacet.catalog</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.efacet.core</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.efacet.edit</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.efacet.editor</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.efacet.metamodel</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.efacet.metamodel.edit</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.efacet.sdk.ui</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.efacet.ui</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.query.java</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.query.java.core</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.query.java.metamodel</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.query.java.sdk.ui</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.query.ocl.core</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.query.ocl.metamodel</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.query.ocl.sdk.ui</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.util.core</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.util.emf.catalog</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.util.emf.core</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.util.emf.ui</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.util.jface.ui</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.util.pde.core</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.util.swt</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.util.ui</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.widgets</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.widgets.celleditors</module>
<module>../plugins/facet/org.eclipse.papyrus.emf.facet.widgets.celleditors.ecore</module
…</modules>
By
<modules>
<module>../plugins/facet</module>
</modules>
And a pom.xml in the directory facet containing all modules
A degraded solution would be to “help” maven by listing module in the order they will be build.
=>Ex : moving the features at the end of the modules list
Regards,
Benoit
De : mdt-papyrus.dev-bounces@xxxxxxxxxxx [mailto:mdt-papyrus.dev-bounces@xxxxxxxxxxx]
De la part de LETAVERNIER Camille
Envoyé : vendredi 13 février 2015 13:36
À : Papyrus Project list
Objet : [PROVENANCE INTERNET] Re: [mdt-papyrus.dev] About tests
Hi,
A single POM would probably help in this case. Invalid builds would probably fail later (Because dependencies from the 4 builds would have to be
resolved before trying to compile anything), but valid builds would probably take less time, so that would help.
Camille
De :
mdt-papyrus.dev-bounces@xxxxxxxxxxx [mailto:mdt-papyrus.dev-bounces@xxxxxxxxxxx]
De la part de Christian W. Damus
Envoyé : vendredi 13 février 2015 13:31
À : Papyrus Project list
Objet : Re: [mdt-papyrus.dev] About tests
I, too, have observed how much time is spent in dependencies. I wonder, is it a maven problem or a Tycho problem? It's an order of magnitude more than the compile step. In any case, do you think any significant time
could be saved by running a single maven build on a pom that is parent to all four of these builds?
On Fri, Feb 13, 2015 at 7:20 AM, LETAVERNIER Camille <Camille.LETAVERNIER@xxxxxx> wrote:
Hi,
The Gerrit job is back to normal. The failures due to the invalid Job configuration
(should) have been retriggered.
Unfortunately, even just compiling tests takes an additional 15-20 minutes (Most of
the time is spent in the dependencies resolution rather than actually compiling). So Gerrit jobs now take a total of 30-40 minutes depending on the server load, and the queue just keeps growing. The total time between a Gerrit contribution and the Hudson validation
may be close to 1 hour now (That’s still much lower than the time between the contribution and the actual committer review however :) )
Hi all,
Bug 459800 [1] has been reported to compile main/tests + extra/tests during the Gerrit
build. I’m currently working on it and started a test build. If I missed anything, other Gerrit builds in progress may receive a -1 from Hudson. If this happens, this may mean that the b job configuration is invalid, rather than your contribution.
I’ll send and update when I’m sure the Gerrit job is properly configured to compile
all tests. Also note that test execution is not yet planed, for performances reasons (Running all tests takes more than an hour, while current Gerrit builds take less than 20 minutes)
[1] 459800: [Releng] Gerrit review builds should compile all tests
https://bugs.eclipse.org/bugs/show_bug.cgi?id=459800
Regards,
Camille
Gerrit compiles Main + Extra, and skips the tests
Camille
De :
mdt-papyrus.dev-bounces@xxxxxxxxxxx [mailto:mdt-papyrus.dev-bounces@xxxxxxxxxxx]
De la part de Christian W. Damus
Envoyé : jeudi 12 février 2015 16:35
À : Papyrus Project list
Objet : Re: [mdt-papyrus.dev] About tests
Reversion is pushed and a new tests build is scheduled.
It seems that our Gerrit review builds not only don’t run the tests, but they also don’t compile them. Do they at least compile the Extras plug-ins? What if
a Gerrit patch includes changes in Extras plug-ins?
On Thu, Feb 12, 2015 at 10:22 AM, LETAVERNIER Camille <Camille.LETAVERNIER@xxxxxx> wrote:
Hi all,
Quick update, since things have changed quite a bit just after I posted the initial report:
-
311 new failures in the test results:
o
The issue is known and being worked on (The SysML BDD Package is broken, causing all the failures)
-
Test build don’t compile anymore
o
Is anyone working on that?
-
Failed to execute goal org.eclipse.tycho:tycho-compiler-plugin:0.22.0:compile (default-compile) on project org.eclipse.papyrus.infra.core.sasheditor.tests: Compilation failure: Compilation
failure:
-
[ERROR] /jobs/genie.modeling.mdt.papyrus/Papyrus-Master-Tests/workspace/tests/junit/plugins/core/org.eclipse.papyrus.infra.core.sasheditor.tests/test/org/eclipse/papyrus/infra/core/sasheditor/pagesmodel/PanelTerm.java:[48]
-
[ERROR] * @see org.eclipse.papyrus.infra.core.sasheditor.IModelObject.sashmodel.query.IQueryTerm#accept(org.eclipse.papyrus.infra.core.sasheditor.IPagesModelVisitor.sashmodel.query.IQueryVisitor,
org.eclipse.emf.ecore.EObject)
There’s a lot of activity on the build server right now, so it may take time for any
fix/commit to be taken into account
Camille
... and of course there is the Gerrit process, with a lot of contributions being delayed
for several weeks.
And one of the main issues in the diagrams is merging the GMFGen/Generated code: we’ve
had a lot of regression due to incorrect merge commits (We don’t have any tool to merge the GMFGen model).
Diagrams are really sensitive, thus the M4/M5 refactorings, which will hopefully help
a lot in the future. But for now... it’s mostly regressions :) (If these modifications hadn’t been pushed, we wouldn’t even be able to work on some diagrams, and especially not in parallel)
BTW, if anyone can figure out what’s wrong with the Master-Tests-Failures job... it
seems the Junit XML file is corrupt, which breaks the “Publish Junit results” and breaks the build. I “suppose” an invalid character appears in one of the 10 000 tests of Papyrus, but couldn’t find which one. So the job is now suspended, and we can’t really
use the test-annotations framework on Master...
Ø
Interesting. Does SVG require a native library that isn’t installed on the build server? We really need to get the tests green (IMHO before M6). Is there something I can do to help? (being quite inexperienced with SVG and the diagrams
generally)
Batik/SVG + Apache Xerces. It looks like the “SaxParser Driver” can’t be found. At least
that’s what the Error log says. I suppose it’s a plug-in dependency issue (I remember we had trouble with inconsistent versions of Batik, because they moved a class from one plug-in to another, and in some configurations, this resulted in NoClassDefFound errors).
It may be another occurrence of this issue, but I haven’t really had the opportunity to look into this so far. I’ve added an explicit dependency to Xerces but this didn’t help; I didn’t go much further.
It may also be a bug in Papyrus (We had some case-sensitivity errors in the images,
which worked fine in Windows/PDE environment, but not in a Linux/Installed eclipse instance), but these errors should be fixed now.
Camille
Ø
(everybody does run the tests locally before they push changes, right? ;-)
Sure :) Remember this issue about results being different on Tycho and JDT ? :)
That’s good. It’s just that sometimes it doesn’t seem like it. I’ve been guilty, myself, through overconfidence, so I make liberal use of smilies. :-)
A lot of the test failures that crop up from time to time are reproducible locally, so they mostly aren’t down to differences in the build environments. But
hopefully we are now a step closer to improving that situation, too.
SVG images seem to be failing on the Hudson/Tycho build for some reason. They caused
a lot of indirect test failures (Took me 6 weeks to figure it out... I guess the over-spammed error log in the test build didn’t help). But these tests were mostly fine, when executed locally.
Interesting. Does SVG require a native library that isn’t installed on the build server? We really need to get the tests green (IMHO before M6). Is there something
I can do to help? (being quite inexperienced with SVG and the diagrams generally)