Re: [eclipse-dev] Parallel builds (Photon+)

[Mickael Istria <mistria@xxxxxxxxxx>]

On Mon, Jul 2, 2018 at 7:35 AM, Duft Markus <Markus.Duft@xxxxxxxxxxxxxxxx> wrote:

Hey,

Hi Markus,

 I have some questions regarding the parallel builder support in Eclipse Photon.

Thanks for your interest in this topic!

Before I give more answers to the specific questions, you may want to watch https://www.youtube.com/watch?v=f8fLuIztku4&list=PLy7t4z5SYNaTSU0ccSoYBG3SxSviFw0-C&index=22 where I tried to present the general concepts and limitations of this implementation, and mention what can, should and will be the next steps. I think this can cover some of your questions.

If I have understood this correctly, there is currently no parallel build support for JDT and PDE builders, right?

Indeed, those builders currently use the WorkspaceRoot scheduling rule, which makes builds exclusive.

The real issue is that at this point of time, we don't yet know whether these builders can safely work in parallel or not, and if they're not, what is the amount and difficulty of required work.

So we may be able to just relax the rule and this would work, or maybe it would crash a lot of things... This uncertain state is where we are right now, and where we need to move away from, by auditing and testing the builders.

 

Even worse, as I have understood [3] and [4], the planned “parallel” support is constructed in a way which would not allow to build two projects of the respective nature (JDT, PDE) in parallel, but only /other/ project types.

That's the defensive approach: change the minimal amount of things to keep a safe state but still enable some progress.

This is not the final goal, the final goal is to have builders thread safe and all running in parallel. This "all related projects rule" can be one interesting increment.

But at EclipseCon France, I had a conversation with Dani who is optimistic about JDT build being almost thread-safe. So we plan to audit this builder and try a more relaxed rule (like project only) that would enable parallel JDT builds. But nothing was concretely started on this pas at the moment, but it's pretty high in our (Red Hat's) todo-list for JDT as it woud have significant positive impact on JDT-LS and cloud deployment.

 Anything I could help with (although I have nearly no time to do so xD)? We have ~750 PDE (which also includes the JDT nature) projects.

That's perfect, and I really like reading this ;)

So if you want to help, then we need pieces of answer to "how far are JDT and PDE builders from being tread-safe". With you use-case, you can try to simply tweak JDT and PDE builders to have a very lax scheduling rule (null or current project), then enable parallel builds over different numbers of threads, then run builds, analyze them and report bugs you see (can be exceptions, unexpected results in output...). Once we have those bugs, we've done a big part of the work, the rest is just to fix them ;)

And when all known bugs are fixed, we can safely change the builders default rule and enable parallel builds by default.

Hope this helps,