| Re: [m2e-dev] Improving m2e performance |
|
FWIW, we were in a very similar situation a while ago (many projects in the workspace, slow build times in m2e compared to plain Maven, and (very) high memory consumption).
We profiled m2e/Maven and managed to get it to a reasonable state.
If you would like to have a look at our changes, or even try and compare them to pristine m2e, they are
here: https://github.com/GEBIT/m2e-core/commits/1.9.1-GEBIT?branch=1.9.1-GEBIT (still
based on m2e 1.9.1, though).
Look for commits by erwint and cpfeiffer.
I'm happy to discuss the changes and would love if the ideas or implementations would somehow get picked up by m2e proper.
Much work was done to avoid unnecessary project bulding. Due to its memory drawback, increasing the project cache was indeed not the cure.
What did help was
- making the project cache more effective by building projects in an order such that often used (parent) projects are built first and remain in the cache
- avoiding project building altogether when not necessary
- trying to reuse MavenProject instances more
What also helped quite a bit was not setting the charset when not necessary and allowing lifecycleMappingMetadata to specify folders which should be ignored by m2e (so not trigger a build), e.g. because only plugin-specific logfiles
or databases are stored there.
We also considerably reduced the per project memory size by deduplicating managedVersionMap and dependencyManagement.dependencies, but this is probably only needed if have deep project hierarchies by using the maven-tiles plugin, for example.
Cheers
Carsten
Von: m2e-dev <m2e-dev-bounces@xxxxxxxxxxx> im Auftrag von Thomas Reinhardt <thomas@xxxxxxxxxxxxx>
Gesendet: Freitag, 28. Mai 2021 23:03 An: m2e-dev@xxxxxxxxxxx <m2e-dev@xxxxxxxxxxx> Betreff: Re: [m2e-dev] Improving m2e performance Thanks Carsten, really a handy view. Didn't know about that. Christoph: I think we can rule out JDT Vivien: Thanks for the hint. I will investigate if increasing the cache improves things. I did try to do a clean test but everything always seemed legit (apart from m2e triggering 2 builds most of the time...). Which is funny because I just opened the "Maven Workspace Build" view, opened more projects to a total of just over 100 and got huge build numbers. Then I tried to do a clean test (committed changes, closed everything etc.) and I was not able to reproduce the numbers. At all. So it seems I'm just holding it wrong :) I tried again: opened the 5 projects I had open before, built everything, fixed the "project not up to date" errors and cleared the "Maven Workspace Build" view. Then I opened the additional projects and voila! Build numbers: 20 14 13 10 9 8 7 (3x) 6 (11x) ... After fixing the "project not up to date" errors: 20 16 13 12 11 9 (3x) 8 (11x) 7 (7x) ... After setting development versions. This is a small eclipse plugin we use to set all the project versions in a BOM project that is imported (dependency with import scope) by all projects. Also, the BOM project version is set in all the projects: 26 22 18 17 16 14 13 (2x) 12 (11x) 11 (7x) ... Something is funny: I really expected the third step (development versions) to break things but the simple opening of the additional projects did far more builds than the clean tests suggested. Any ideas if this behavior could be explained by the cache issue Vivien mentioned? -Thomas On 28/05/2021 17:05, Carsten Pfeiffer wrote: > There is the very handy "Maven Workspace Build" view in eclipse, that > can tell you how often builds are triggered, and by which resources. > Just open the view and deactivate the "Suspend" button. > > Cheers > Carsten > > ------------------------------------------------------------------------ > *Von:* m2e-dev <m2e-dev-bounces@xxxxxxxxxxx> im Auftrag von Christoph > Läubrich <laeubi@xxxxxxxxxxxxxx> > *Gesendet:* Freitag, 28. Mai 2021 15:36 > *An:* m2e-dev@xxxxxxxxxxx <m2e-dev@xxxxxxxxxxx> > *Betreff:* Re: [m2e-dev] Improving m2e performance > Have you tried to attach a profile (e.g. JProfiler) to get some feeling > where the bottlenecks are? Are you sure that m2e is actually triggering > the builds (e.g. do you see code getting generated repeatably) or is it > more JDT itself rebuilding because of changed code parts? > > > Am 28.05.21 um 14:53 schrieb Thomas Reinhardt: >> >> Hello List! >> >> TLDR: >> We have a large maven build (200+ projects) which is very slow to build >> in eclipse. I have built tools and procedures to be able to work with >> only a few projects open comfortably. So this is not a cry for help. I >> want to help improve m2e and I think this might be an interesting use >> case/benchmark. >> >> Long version: >> As stated above our whole product consists of over 200 different >> projects. A full maven build (without tests) on my laptop takes about 15 >> minutes on the command line ("mvn -DskipTests clean package"). In >> eclipse the build often does not finish at all or needs hours. I can >> speed up things when I open projects in groups and wait for the build to >> finish. Even with that workaround it might take me 2 hours until I have >> opened all projects. This seems very wrong as eclipse should at least >> roughly match the command line build speed. >> >> To be clear: in my normal daily work I don't have to open all projects. >> I only ever do that for complex refactorings and things like that. >> Normal daily performance is no problem. >> Having said that: I know from the "all projects open" use case that m2e >> does have some issues and builds projects more than once. It is just not >> noticeable when I only have a dozen projects open. >> >> So: I am offering to do any tests some of you might want to do me, give >> remote access or even send you the code. The last one might need some >> work on my side to remove non-public dependencies, maven plugins etc but >> that should not be a problem. >> >> I did try to cut the problem down to a few single offending projects but >> performance just gradually improved/worsened (depending on your >> viewpoint). So I think it is really down to m2e building projects too >> often. Also, the builds are (in my opinion) not very complex: we do >> generate some code from xsd and there is one in-house built maven plugin >> that generates hibernate code from an xml file. Also, this is a javaee >> product (that ultimately generates an ear file and a standalone client) >> and we use m2e-wtp of course. >> Also, I tried to debug m2e myself but I quickly realized that I don't >> know enough of the eclipse build model to make any sense of the code. >> >> This is arguably a complex setup and hard to debug but it might just >> make m2e so much better. Please tell me if there is interest to >> investigate the performance issue. >> >> >> >> PS: >> For completeness my system specs: >> debian 10, 32GB ram, intel core i7-8550U, SSD, eclipse 2021-03 >> m2e 1.17.2.20210219-1922 >> >> >> >> _______________________________________________ >> m2e-dev mailing list >> m2e-dev@xxxxxxxxxxx >> To unsubscribe from this list, visit >> https://www.eclipse.org/mailman/listinfo/m2e-dev > <https://www.eclipse.org/mailman/listinfo/m2e-dev> > _______________________________________________ > m2e-dev mailing list > m2e-dev@xxxxxxxxxxx > To unsubscribe from this list, visit > https://www.eclipse.org/mailman/listinfo/m2e-dev > <https://www.eclipse.org/mailman/listinfo/m2e-dev> > > _______________________________________________ > m2e-dev mailing list > m2e-dev@xxxxxxxxxxx > To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/m2e-dev > _______________________________________________ m2e-dev mailing list m2e-dev@xxxxxxxxxxx To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/m2e-dev |
Attachment:
smime.p7s
Description: S/MIME cryptographic signature