| Re: [papyrus-rt-dev] Multiple Papyrus-RT-Master-Product builds for each merged commit |
Hi Peter, Hi all.
There is a small confusion, and I’m getting finally confused as well.
In Hudson configuration, for an integration Job, it is not possible to trigger a Papyrus-RT-Master-XXXX depending on a specific modification.
This is only possible on the Gerrit Event trigger .
So for example, if a change is made on a tooling file, then only the tooling Gerrit job will be triggered.
But once this change has been merged to the master branch, this is the Papyrus-RT-Master-Core (which is the starting point of our integration hierarchy) that will check every 15 min if there is any change made on the Git of “org.eclipse.papyrus-rt”.
If ANY modification has been merged (even on releng, pom, or README file), it builds the Core Component and then triggers Tooling and Codegen Master jobs, and once both are completed, the Product build the releng part, thanks to the join configuration.
So the Papyrus-RT-Master-Tooling job should normally not trigger on is own. Except if manually done.
Do you have the Tooling build reference so I can check if a manual build has been done ?
Regards
Céline
De : papyrus-rt-dev-bounces@
eclipse.org [mailto:papyrus-rt-dev-bounces@xxxxxxxxxxx ] De la part de Peter Cigéhn
Envoyé : mercredi 12 octobre 2016 13:35
À : papyrus-rt developer discussions <papyrus-rt-dev@xxxxxxxxxxx>
Objet : Re: [papyrus-rt-dev] Multiple Papyrus-RT-Master-Product builds for each merged commit
Hi,
It looks like we have exactly the case where only the tooling build have been triggered. Ansgar merged the change for the final 0.8 Bugzilla, https://bugs.eclipse.org/bugs/
show_bug.cgi?id=504059 , and to me it looks like only the tooling component has been triggered by this. Nothing seem to be triggering the final product build... And I myself do no longer have the access right to manually trigger the product build myself (as I had for some of the old builds).
Could someone ensure that the product build is triggered (manually?) so that I can test this change using the tester setup?
/Peter Cigéhn
On 11 October 2016 at 20:33, Christian Damus <give.a.damus@xxxxxxxxx> wrote:
Thanks, Céline.
The options that you outline are clear. I guess we’re really fighting the Hudson architecture on this one, eh? Our scenario strikes me as so perfectly common that I’m really surprised it isn’t better accounted for by the available tools. Is there something we’ve overlooked? Or maybe what we’re trying to do with component builds isn’t so common, and that for good reason?
I honestly don’t know which of these options I would recommend over the others. All three have characteristics that appeal to me.
In any case, I think that perhaps we should put any plans to further decompose the builds on ice until we get a conclusion to this question.
cW
On 11 October, 2016 at 11:30:15, Céline JANSSENS (celine.janssens@xxxxxxxxxxx) wrote:
Hi Christian,
No it won’t trigger the Product if only Tooling is impacted.
This is the regression of the join method.
If I keep the cascading from Tooling and from Codegen to Product, in addition with the Join trigger, Product will be triggered 3 times ;s
So there are 3 solutions with their inconvenience:
1) We keep it as previously => Product triggered twice if Core is impacted, once if Tooling or Codegen, and all combinations if changes impact several components.
2) We keep the join and trigger the Core on every modification => 3 Components are built even if not necessary but only once
3) We cascading Core > Tooling > Codegen > Product => If only Tooling is impacted, this configuration builds Codegen even if not necessary and all combinations if changes impact several components.
So if in most of the case we modify only one component by Change or every 15 minutes, the 3d solution is the more efficient.
Otherwise, if there are several components impacted most of time, the second solution is the one we should put in place.
I don’t know a solution meeting 100% the requirements.
Regards
Céline
De : papyrus-rt-dev-bounces@
eclipse.org [mailto:papyrus-rt-dev-bounces@xxxxxxxxxxx ] De la part de Christian Damus
Envoyé : mardi 11 octobre 2016 17:09
À : papyrus-rt developer discussions <papyrus-rt-dev@xxxxxxxxxxx>
Objet : Re: [papyrus-rt-dev] Multiple Papyrus-RT-Master-Product builds for each merged commit
Hi, Céline,
This looks very cool, thanks!
One question: I see that all of the configuration is now on the Core job, with the new join trigger. Will the Product build still be run if only, for example, the Tooling build is triggered by a commit that makes only tooling changes? Or will Product now only build after both Codegen and Tooling builds are triggered by a Core build (from a commit that changes core projects)?
Thanks,
Christian
On 11 October, 2016 at 10:48:03, Céline JANSSENS (celine.janssens@xxxxxxxxxxx) wrote:
Hi All,
The Join trigger plugin being in place, I configured the different build to avoid to trigger Papyrus-RT-Master-Product twice.
Papyrus-RT-Master-Product is now triggered when both Tooling and Codegen are complete
More information about the Join trigger plugin installed on the HIPP available here [1]
Let me know if you meet any issues.
Regards
Céline
[1] : https://wiki.jenkins-ci.org/
display/JENKINS/Join+Plugin
De : papyrus-rt-dev-bounces@
eclipse.org [mailto:papyrus-rt-dev-bounces@xxxxxxxxxxx ] De la part de SCHNEKENBURGER Remi 211865
Envoyé : mercredi 5 octobre 2016 11:46
À : papyrus-rt developer discussions <papyrus-rt-dev@xxxxxxxxxxx>
Objet : Re: [papyrus-rt-dev] Multiple Papyrus-RT-Master-Product builds for each merged commit
Hi team,
Yes, I am (finally!) back from vacation.
I installed the “join” plugin, as this seems to be the most suitable plugin for our needs. I also installed a utility to manage job configs (a kind of version control management for job configurations)
Hudson will be restarted when current jobs are finished, so these plugins will be activated.
Regards,
Rémi
------------------------------
-------------------------
Rémi SCHNEKENBURGER
CEA Saclay Nano-INNOV
Institut CARNOT CEA LIST
De : papyrus-rt-dev-bounces@
eclipse.org [mailto:papyrus-rt-dev-bounces@xxxxxxxxxxx ] De la part de Céline JANSSENS
Envoyé : mercredi 5 octobre 2016 09:40
À : papyrus-rt developer discussions <papyrus-rt-dev@xxxxxxxxxxx>
Cc : remi.schnekenbuger@xxxxxx
Objet : Re: [papyrus-rt-dev] Multiple Papyrus-RT-Master-Product builds for each merged commit
Hello Chrsitian,
That is Remi that has such access rights, for the HIPP.
He is back from his vacations, so do not hesitate to see with him ;)
Your solution a) seems a good point. I didn’t know that it exists. Well done.
Regards
Céline
De : papyrus-rt-dev-bounces@
eclipse.org [mailto:papyrus-rt-dev-bounces@xxxxxxxxxxx ] De la part de Christian Damus
Envoyé : mardi 4 octobre 2016 21:12
À : papyrus-rt developer discussions <papyrus-rt-dev@xxxxxxxxxxx>
Objet : Re: [papyrus-rt-dev] Multiple Papyrus-RT-Master-Product builds for each merged commit
Hi,
Yes, it’s a problem that we have the downstream Product build running once for each completion of its multiple upstreams.
There are plug-ins that can help this. Hudson Plugin Central has at least two that address this situation:
· join: triggers a job after a group of other jobs finish. It’s the simplest implementation, and I think would be suitable for our case (I have seen reports that it doesn’t work well when parallel job chains to be joined are more than 1 job long)
· build-pipeline-plugin: a visual configuration of complex pipelines of jobs
I don’t have admin privilege on our HIPP to see which, if any, of these plug-ins might already be available for us to use. Nor do I know the process for requesting to install new plug-ins, if that’s a thing that we can do for our own HIPP should we decide to do so.
And, yes, a change in core should not trigger both the Core and Tooling builds but Core only, because it will kick off Tooling as a downstream anyways. That much is easy to address; I can take care of that.
Hi,
I am not sure I understand the proposed change. Currently the product build is triggered by the component builds, core, tooling and so on. I assumed that the reason for the product build to be triggered by the component builds was due to that the product build used the output from the component builds in some way.
But now you are saying that the product build can be triggered "on its own", which I interpret as if it does not really depend on the component builds have been run at all. Is that correct? Then what is the purpose of the individual component builds? I am bit confused here...
And when you say each 30 minutes, I assume that you mean that the product build should check if there has been an SCM change, i.e. a commit being merged, each 30 minutes. But why should that be check only be made each 30 minutes? That I assume could be checked each 15 minutes as for the component builds. Otherwise you can theoretically have to wait for 30 minutes before the build is triggered (in case the merge is committed right after the last SCM poll) and then the build that takes 17-20 minutes, i.e. up to 50 minutes.
Sure, that is much better than the current situation where you need to wait up to 15 minutes for the SCM poll on the component builds, then build core and tooling in parallell (I assume a few minutes), then build product for up to 24 minutes, in parallel with the second tooling build,, and then the second product build (triggered by the second tooling build) for 24 minutes more, which adds up to more than an hour. Still not sure if I can use the first product build or if I need to wait for the second one.
I thought that the triggering of builds could be made a bit more intelligent, i.e. if it is concluded that a commit impacts both core and tooling, then skip triggering tooling and only trigger core, since you know that core in its turn will trigger the build of tooling. That would avoid the double builds of tooling and product.
/Peter Cigéhn
On 4 October 2016 at 17:45, Céline JANSSENS <celine.janssens@xxxxxxxxxxx> wrote:
Hi Peter,
As far as I remember, The Core build triggers Tooling and Codegen jobs. Codegen and Tooling jobs both trigger the Product job.
So you are right, every time a modification is done on the core, then tooling and codegen jobs are triggered. And both trigger the Product job.
It has been whished that the Product should be up to date at least every 15 minutes. That is why, Core, Tooling and Codegen Build check for modification every 15 minutes.
Product job is based on Core, Tooling and Codegen sub-repos as it builds the “releng” Target platform.
A solution would be not to trigger the product directly from the Components builds, but on a regular basis, 30 minutes seem reasonable.
In addition, the RCPTT is being extract from the Product Build to decrease the time of the Product Build to 17-20 minutes.
Any remarks on this suggestion ?
Regards
Céline
De : papyrus-rt-dev-bounces@
eclipse.org [mailto:papyrus-rt-dev-bounces@xxxxxxxxxxx ] De la part de Peter Cigéhn
Envoyé : mardi 4 octobre 2016 17:19
À : papyrus-rt developer discussions <papyrus-rt-dev@xxxxxxxxxxx>
Objet : [papyrus-rt-dev] Multiple Papyrus-RT-Master-Product builds for each merged commit
Hi,
I have a question regarding the new build system. I find a bit confusing knowing when it is safe to start upgrading and testing a merged commit. Almost all the latest commits that I have been merged and which I wanted to test, and for which I then I have checked the builds when they are finished, have been triggering the final Papyrus-RT-Master-Product to be built (at least) twice.
I guess that it is related to that a commit contains changes, e.g. for both core and tooling, which triggers both these builds (in parallell). When these are finished, they trigger the final product build. And then when the core is finished that triggers yet another tooling build, which in its turn triggers a product build again. In more or less all cases I have seen, the second product build has gotten stuck in the queue also since you cannot build two product builds in parallell.
Is it safe to test already after the first product build have finished, or do I always have to wait for the second one to finish? More or less always having to wait for two product builds feels pretty time consuming (since each takes up to half an hour to complete).
Is the setup of the build system really correct? Why is both core and tooling triggered at the first step? If you only trigger building core, which then triggers tooling, which finally triggers product, then you don't have tooling and product building twice.
Any comments? Is this just how it is supposed to work? And that I (to be on the safe side) always needs to wait for the second (or whatever number the final will be) build before I can upgrade safely?
/Peter Cigéhn
_______________________________________________ papyrus-rt-dev@xxxxxxxxxxx
papyrus-rt-dev mailing list
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/papyrus-rt- dev
______________________________
_________________ papyrus-rt-dev@xxxxxxxxxxx
papyrus-rt-dev mailing list
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/papyrus-rt- dev ______________________________
_________________ papyrus-rt-dev@xxxxxxxxxxx
papyrus-rt-dev mailing list
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/papyrus-rt- dev ______________________________
_________________ papyrus-rt-dev@xxxxxxxxxxx
papyrus-rt-dev mailing list
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/papyrus-rt- dev
_______________________________________________ papyrus-rt-dev@xxxxxxxxxxx
papyrus-rt-dev mailing list
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/papyrus-rt- dev
_______________________________________________
papyrus-rt-dev mailing list
papyrus-rt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/papyrus-rt- dev
