Re: [dash-dev] Streamlining the Athena promote process (was Re: [gmf-dev] Head up: last night's build failure was a real failure in org.eclipse.gmf.xpand.migration)
Dude, "creating a cron job" is like 5 mins of work. Copy, paste, issue command `crontab -e`, set polling time/interval, and vavoom, done. Hardly a high barrier. Getting yourself a bash account on build.eclipse.org takes longer (and that only requires filing a bug and waiting). :)
As to a Hudson Plugin project in Dash, sure... cuz it's that or at SF or Google Code or wherever else code is hosted. If you want to jump thru the hoops to make it happen, by all means do.
As to slaves, one workaround there is to enable the 'publishing plugin' which allows the output of a build to be pushed into another Hudson instance (then poll that instance for updates); alternatively, if you tie your job to a single slave, it's just as effective as being tied to master: poll the slave/master on which your build runs.
At some point, when we have the option to run across multiple slaves/platforms, the only requirements will therefore be that the crontab script poll all the appropriate places and that the slaves all support ssh/scp/rsync in order to be pollable. Or failing that, ftp/http, I suppose.
Well, the SCP plugin is Open Source, so I'm sure if some enterprising
committer wanted to modify it to allow more configuration options one
could do so. These types of concerns are why I think Dash may need to
have a Hudson Plugin subproject. This way you can address the eclispe
specific environment concerns.
Nick, how does your Batch Task/Cron idea work in a distributed system
when Slave machines? What about slave machines that aren't Linux
based? How does it work with Windows based machines? Some things to
keep in mind.
I just think having to make users create cron jobs is just another
barrier that isn't necessarily necessary. I'd still prefer something
that was more integrated instead of doing batch magic which can be
still as insecure as SCP rights. Just takes one wrong batch job
executing at the wrong access level to mess things up.
Dave
On 01/29/2010 06:24 AM, Denis Roy wrote:
+1
Cron jobs that poll are not nearly as elegant, but they do allow for
better security.
D.
On 01/28/2010 10:59 PM, Nick Boldt wrote:
Problem
with SCP plugin is that it uses global scp access rights to move files;
it doesn't seem to be able to key in to the LDAP database so that I can
push files to nickb@xxxxxxxxxxxxxxx:~/downloads/,
for example.
My workaround is to have Hudson write to its own workspace (eg., create
a .lock file that says "job foo's build #34 is ready to promote") and
then have nickb's crontab look for that file every minute or two; if
found, it can perform the publish and either delete the .lock file, or
record somehow that it's already pushed that build so as to not do so
again... unless the Batch Task is kicked again to refresh the file.
So, like with the signing genie, we'd have a process that can pass
information between users@dev.eclipse
and the shared user hudsonBuild
in order to allow uease of use AND security.
N
On 01/28/2010 05:02 PM, David Carver wrote:
If we are concerned about giving Hudson
write
access to the downloads
directory, then we might want to consider the Batch Tasks Plugin that
Nick has suggested. You could run an ftp/scp command from there and ftp
the files over to the download server. At least this way it is running
under specific committer ids, which would then limit it to specific
directories instead of the whole thing.
There is also the SCP plugin for Hudson which allows for a secure
connection and copying of files.
It would be nice to be able to tag cvs projects as part of a hudson
build. Today, when we run our integration builds, we tag our builder
and our map file project so that the build is reproducible. Once we
have test hardware at the foundation and can run our builds on the
hudson install on build.eclipse.org, the only way to do this is to
have cronjob entries that correspond the time our build starts, which
isn't 100% foolproof. That being said, I totally understand Denis'
concern with the Hudson user having write access to the larger
download and source filesystem. Not good from a security perspective.
Subject
Re: [dash-dev] Streamlining the Athena promote process (was Re:
[gmf-dev] Head up: last night's build failure was a real failure in
org.eclipse.gmf.xpand.migration)
Promotion should only require write access to to the various download
area folders for the projects.
It wouldn't necessarily have to have tagging priviledges to
CVS/SVN/Git as it stands right now all projects use MAP files for this.
Dave
On 01/28/2010 08:24 AM, Denis Roy wrote:
For Hudson to do the promotion itself, I assume the hudsonBuild user
would need write access to most, if not all of the downloads area. If
the promotion process involves tagging or committing to CVS, then
hudsonBuild would need commit access to CVS.
Ick.
On 01/28/2010 11:11 AM, David Carver wrote:
Nick, Hudson already has plugins that would allow the promotion. The
problem is that the user that Hudson runs under does not have rights
to the download server to be able to push the necessary bits. We have
the necessary plugins installed in Hudson, just don't have the access
rights.
Instead of jumping through work arounds and hoops we really need to
address the problem at hand. Hudson should have necessary access
rights to be able to do the promotion itself.
Dave
On 01/27/2010 11:43 PM, Nick Boldt wrote:
Theoretically, it should be posible to create a hudson job that would
be used simply to promote the latest clean builds by flipping a bit to
notify a listening process (namely a crontab script) that it's time to
promote.
But does having a button on a webpage really make publishing easier
than ssh'ing to a server and running a script? The same code that the
crontab runs can be run on demand, and you could wrap that line of
code w/ a wrapper script such that all you'd need to do is ssh to
build.eclipse and run "~/p" to push your bits. Only marginally easier
than logging in to build.eclipse.org to push a button.
Would an Eclipse plugin would be perhaps a better idea? Perhaps
something graphically modeled (hint: GMF? Zest?), with not just a
button but a view of your latest builds in Hudson and their test
results?
Or, perhaps a Hudson plugin is better, such that in addition to the
build button we already have, we could also have a promote button?
N
On 01/25/2010 09:54 AM, Anthony Hunter wrote:
Hi Nick,
With GMF and the common modeling build, have a web page with two
buttons, build and promote
With the Athena build, have a web page with a build button, but no
promote yet. You last communicated promote must be manually ran from
the
command line or cron.
>From my point of view, the lack of a quick easy promote have Athena
a
hard sell.
To be honest however, I have not had any time to look further that
Athena for GEF.
Once GEF and EMF have fully moved, the rest of the modeling stack (and
GMF) should have no excuse to move.
And the goal should be / is for all modeling projects to share the same
build technology.
Cheers...
Anthony
--
Anthony Hunter _mailto:anthonyh@xxxxxx.com_
Software Development Manager
IBM Rational Software: Aurora / Modeling Tools
Phone: 613-270-4613
Inactive hide details for Nick Boldt ---2010/01/25 12:54:34 AM---If
only
there was a better way... like building against updateNick Boldt
---2010/01/25 12:54:34 AM---If only there was a better way... like
building against update sites
With Galileo SR2 nearly complete, does GMF plan to move to
Athena/Hudson during the Helios cycle? If not, what blocks you? I'd
like a list of blocking requirements so I can better prioritize Athena
TODOs and get it into a form that will meet more (if not all) of your
needs.
BTW, the tagandrelease system works just as well for Athena builds as
for Modeling builds; the only difference is that you have full control
over it (ie., no CVS commit issues) and would run it on build.eclipse.org instead of modeling.eclipse.org.
Cheers,
Nick
On Wed, Jan 20, 2010 at 9:55 AM, Anthony Hunter
_<anthonyh@xxxxxxxxxx>_ <mailto:anthonyh@xxxxxxxxxx>
wrote:
> Hi Team,
>
> Releng update. I am trying to restore
org.eclipse.releng.tools.tagandrelease
> for GMF. It runs from cron on modeling.eclipse.org. It checks CVS
for new
> changes, tags and releases the new code in the gmf map files and
then
runs a
> build. This would be a completely hands off run a nightly
Integration
build
> when a change is committed.
>
> Everything is now working fine, but the dependencies calculator
does
not
> work when firing a new GMF build.
>
> This is he explanation for the constant failing GMF builds, you
need
to pick
> the correct dependencies as well as using the linux-gtk-x86_64
Eclipse SDK.
>
> This is _https://bugs.eclipse.org/bugs/show_bug.cgi?id=299802_
that
we have
> yet to fix.
>
> Last night I ran a integration build manually selecting what I
think
was the
> latest dependencies:
> _
__http://modeling.eclipse.org/modeling/gmf/gmf/downloads/drops/2.3.0/I201001192155/buildlog.txt_
Denis Roy