Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [dash-dev] Streamlining the Athena promote process (was Re: [gmf-dev] Head up: last night's build failure was a real failure in org.eclipse.gmf.xpand.migration)
Denis is there a place where the security requirements are documented for moving files between systems at eclipse? The reason I'm asking is that it would help me come up with an integrated solution (i.e. possibly updating the SCP plugin for specific user logins). Let me see what type of license the SCP plugin at Hudson is under. I can start with that, as a basis for a possible DASH project for Hudson specific plugins and see what happens from there. I'm fine for the Cron job idea, as long as it's documented. 

Dave


Nick Boldt wrote:
Dude, "creating a cron job" is like 5 mins of work. Copy, paste, issue command `crontab -e`, set polling time/interval, and vavoom, done. Hardly a high barrier. Getting yourself a bash account on build.eclipse.org <http://build.eclipse.org> takes longer (and that only requires filing a bug and waiting). :)
As to a Hudson Plugin project in Dash, sure... cuz it's that or at SF or Google Code or wherever else code is hosted. If you want to jump thru the hoops to make it happen, by all means do.
As to slaves, one workaround there is to enable the 'publishing plugin' which allows the output of a build to be pushed into another Hudson instance (then poll that instance for updates); alternatively, if you tie your job to a single slave, it's just as effective as being tied to master: poll the slave/master on which your build runs.
At some point, when we have the option to run across multiple slaves/platforms, the only requirements will therefore be that the crontab script poll all the appropriate places and that the slaves all support ssh/scp/rsync in order to be pollable. Or failing that, ftp/http, I suppose. 

N
On Fri, Jan 29, 2010 at 10:51 AM, David Carver <d_a_carver@xxxxxxxxx <mailto:d_a_carver@xxxxxxxxx>> wrote: 

    Well, the SCP plugin is Open Source, so I'm sure if some
    enterprising committer wanted to modify it to allow more
    configuration options one could do so.   These types of concerns
are why I think Dash may need to have a Hudson Plugin subproject. This way you can address the eclispe specific environment concerns. 

    Nick, how does your Batch Task/Cron idea work in a distributed
    system when Slave machines?   What about slave machines that
    aren't Linux based?   How does it work with Windows based
    machines?   Some things to keep in mind.

    I just think having to make users create cron jobs is just another
    barrier that isn't necessarily necessary.   I'd still prefer
    something that was more integrated instead of doing batch magic
    which can be still as insecure as SCP rights.   Just takes one
    wrong batch job executing at the wrong access level to mess things up.

    Dave


    On 01/29/2010 06:24 AM, Denis Roy wrote:

    +1

    Cron jobs that poll are not nearly as elegant, but they do allow
    for better security.

    D.

    On 01/28/2010 10:59 PM, Nick Boldt wrote:

    Problem with SCP plugin is that it uses global scp access rights
    to move files; it doesn't seem to be able to key in to the LDAP
    database so that I can push files to
    nickb@xxxxxxxxxxxxxxx:~/downloads/
    <mailto:nickb@xxxxxxxxxxxxxxx:%7E/downloads/>, for example.

    My workaround is to have Hudson write to its own workspace (eg.,
    create a .lock file that says "job foo's build #34 is ready to
    promote") and then have nickb's crontab look for that file every
    minute or two; if found, it can perform the publish and either
    delete the .lock file, or record somehow that it's already
    pushed that build so as to not do so again... unless the Batch
    Task is kicked again to refresh the file.

    So, like with the signing genie, we'd have a process that can
    pass information between users@dev.eclipse
    <mailto:users@dev.eclipse> and the shared user hudsonBuild in
    order to allow uease of use AND security.

    N

    On 01/28/2010 05:02 PM, David Carver wrote:

    If we are concerned about giving Hudson write access to the
    downloads
    directory, then we might want to consider the Batch Tasks
    Plugin that
    Nick has suggested. You could run an ftp/scp command from there
    and ftp
    the files over to the download server. At least this way it is
    running
    under specific committer ids, which would then limit it to
    specific
    directories instead of the whole thing.

    There is also the SCP plugin for Hudson which allows for a secure
    connection and copying of files.

    http://wiki.hudson-ci.org/display/HUDSON/SCP+plugin

    Dave

    On 01/28/2010 12:41 PM, Kim Moir wrote:


    It would be nice to be able to tag cvs projects as part of a
    hudson
    build. Today, when we run our integration builds, we tag our
    builder
    and our map file project so that the build is reproducible.
    Once we
    have test hardware at the foundation and can run our builds on
    the
    hudson install on build.eclipse.org
    <http://build.eclipse.org>, the only way to do this is to
    have cronjob entries that correspond the time our build
    starts, which
    isn't 100% foolproof. That being said, I totally understand
    Denis'
    concern with the Hudson user having write access to the larger
    download and source filesystem. Not good from a security
    perspective.

    Kim




    *David Carver <d_a_carver@xxxxxxxxx>
    <mailto:d_a_carver@xxxxxxxxx>*
    Sent by: dash-dev-bounces@xxxxxxxxxxx
    <mailto:dash-dev-bounces@xxxxxxxxxxx>

    01/28/2010 11:54 AM
    Please respond to
    Tools for Committer Community <dash-dev@xxxxxxxxxxx>
    <mailto:dash-dev@xxxxxxxxxxx>
To 
        Tools for Committer Community <dash-dev@xxxxxxxxxxx>
    <mailto:dash-dev@xxxxxxxxxxx>
    cc
Subject 
        Re: [dash-dev] Streamlining the Athena promote process
    (was Re:
    [gmf-dev] Head up: last night's build failure was a real
    failure in
    org.eclipse.gmf.xpand.migration)








    Promotion should only require write access to to the various
    download
    area folders for the projects.

    It wouldn't necessarily have to have tagging priviledges to
    CVS/SVN/Git as it stands right now all projects use MAP files
    for this.

    Dave

    On 01/28/2010 08:24 AM, Denis Roy wrote:
    For Hudson to do the promotion itself, I assume the
    hudsonBuild user
    would need write access to most, if not all of the downloads
    area. If
    the promotion process involves tagging or committing to CVS, then
    hudsonBuild would need commit access to CVS.

    Ick.




    On 01/28/2010 11:11 AM, David Carver wrote:
    Nick, Hudson already has plugins that would allow the
    promotion. The
    problem is that the user that Hudson runs under does not have
    rights
    to the download server to be able to push the necessary bits.
    We have
    the necessary plugins installed in Hudson, just don't have the
    access
    rights.

    Instead of jumping through work arounds and hoops we really
    need to
    address the problem at hand. Hudson should have necessary access
    rights to be able to do the promotion itself.

    Dave

    On 01/27/2010 11:43 PM, Nick Boldt wrote:
    Theoretically, it should be posible to create a hudson job
    that would
    be used simply to promote the latest clean builds by flipping
    a bit to
    notify a listening process (namely a crontab script) that it's
    time to
    promote.

    But does having a button on a webpage really make publishing
    easier
    than ssh'ing to a server and running a script? The same code
    that the
    crontab runs can be run on demand, and you could wrap that
    line of
    code w/ a wrapper script such that all you'd need to do is ssh to
    build.eclipse and run "~/p" to push your bits. Only marginally
    easier
    than logging in to build.eclipse.org
    <http://build.eclipse.org> to push a button.

    Would an Eclipse plugin would be perhaps a better idea? Perhaps
    something graphically modeled (hint: GMF? Zest?), with not just a
    button but a view of your latest builds in Hudson and their
    test results?

    Or, perhaps a Hudson plugin is better, such that in addition
    to the
    build button we already have, we could also have a promote
    button?

    N

    On 01/25/2010 09:54 AM, Anthony Hunter wrote:
    Hi Nick,

    With GMF and the common modeling build, have a web page with two
    buttons, build and promote
    With the Athena build, have a web page with a build button,
    but no
    promote yet. You last communicated promote must be manually
    ran from the
    command line or cron.

    >From my point of view, the lack of a quick easy promote have
    Athena a
    hard sell.

    To be honest however, I have not had any time to look further
    that
    Athena for GEF.

    Once GEF and EMF have fully moved, the rest of the modeling
    stack (and
    GMF) should have no excuse to move.

    And the goal should be / is for all modeling projects to share
    the same
    build technology.

    Cheers...
    Anthony
-- Anthony Hunter _mailto:anthonyh@xxxxxx.com_ 
    Software Development Manager
    IBM Rational Software: Aurora / Modeling Tools
    Phone: 613-270-4613


    Inactive hide details for Nick Boldt ---2010/01/25 12:54:34
    AM---If only
    there was a better way... like building against updateNick Boldt
    ---2010/01/25 12:54:34 AM---If only there was a better way...
    like
    building against update sites


    From:
    Nick Boldt _<nickboldt@xxxxxxxxx>
    <mailto:nickboldt@xxxxxxxxx>_ <mailto:nickboldt@xxxxxxxxx>

    To:
    "GMF Project developer discussions." _<gmf-dev@xxxxxxxxxxx>
    <mailto:gmf-dev@xxxxxxxxxxx>_
    <mailto:gmf-dev@xxxxxxxxxxx>

    Date:
    2010/01/25 12:54 AM

    Subject:
    Re: [gmf-dev] Head up: last night's build failure was a real
    failure in
    org.eclipse.gmf.xpand.migration

    ------------------------------------------------------------------------




    If only there was a better way... like building against update
    sites
    instead of (or in addition to) SDK zips...

    oh, wait... there is!
    _
    __http://wiki.eclipse.org/Common_Build_Infrastructure/Defining_Binary_Dependencies_



    With Galileo SR2 nearly complete, does GMF plan to move to
    Athena/Hudson during the Helios cycle? If not, what blocks
    you? I'd
    like a list of blocking requirements so I can better
    prioritize Athena
    TODOs and get it into a form that will meet more (if not all)
    of your
    needs.

    BTW, the tagandrelease system works just as well for Athena
    builds as
    for Modeling builds; the only difference is that you have full
    control
    over it (ie., no CVS commit issues) and would run it on
    build.eclipse.org <http://build.eclipse.org> instead of
    modeling.eclipse.org <http://modeling.eclipse.org>.

    Cheers,

    Nick

    On Wed, Jan 20, 2010 at 9:55 AM, Anthony Hunter
    _<anthonyh@xxxxxxxxxx> <mailto:anthonyh@xxxxxxxxxx>_
    <mailto:anthonyh@xxxxxxxxxx> wrote:
    > Hi Team,
    >
    > Releng update. I am trying to restore
    org.eclipse.releng.tools.tagandrelease
    > for GMF. It runs from cron on modeling.eclipse.org
    <http://modeling.eclipse.org>. It checks CVS
    for new
    > changes, tags and releases the new code in the gmf map files
    and then
    runs a
    > build. This would be a completely hands off run a nightly
    Integration
    build
    > when a change is committed.
    >
    > Everything is now working fine, but the dependencies
    calculator does
    not
    > work when firing a new GMF build.
    >
    > This is he explanation for the constant failing GMF builds,
    you need
    to pick
    > the correct dependencies as well as using the linux-gtk-x86_64
    Eclipse SDK.
    >
    > This is
    _https://bugs.eclipse.org/bugs/show_bug.cgi?id=299802_ that
    we have
    > yet to fix.
    >
    > Last night I ran a integration build manually selecting what
    I think
    was the
    > latest dependencies:
    > _
    __http://modeling.eclipse.org/modeling/gmf/gmf/downloads/drops/2.3.0/I201001192155/buildlog.txt_


    >
    > It has failed in org.eclipse.gmf.xpand.migration
    >
    > Can the tooling confirm?
    >
    > The dependencies were:
    > -URL
    > _
    __http://download.eclipse.org/eclipse/downloads/drops/I20100119-0800/eclipse-SDK-I20100119-0800-linux-gtk-x86_64.tar.gz_


    > -URL
    > _
    __http://download.eclipse.org/modeling/emf/emf/downloads/drops/2.6.0/I201001101746/emf-xsd-SDK-I201001101746.zip_


    > -URL
    > _
    __http://download.eclipse.org/modeling/mdt/uml2/downloads/drops/3.1.0/S200912141514/mdt-uml2-SDK-3.1.0M4.zip_


    > -URL
    > _
    __http://download.eclipse.org/tools/orbit/downloads/drops/R20090825191606/orbitBundles-R20090825191606.map_


    > -URL
    > _
    __http://modeling.eclipse.org/modeling/emf/query/downloads/drops/1.4.0/S200912161005/emf-query-SDK-1.4.0M4.zip_


    > -URL
    > _
    __http://modeling.eclipse.org/modeling/emf/transaction/downloads/drops/1.4.0/S200912161108/emf-transaction-SDK-1.4.0M4.zip_


    > -URL
    > _
    __http://modeling.eclipse.org/modeling/emf/validation/downloads/drops/1.4.0/S200912161043/emf-validation-SDK-1.4.0M4.zip_


    > -URL
    > _
    __http://download.eclipse.org/tools/gef/downloads/drops/3.6.0/I201001151504/GEF-SDK-I201001151504.zip_


    > -URL
    > _
    __http://download.eclipse.org/modeling/m2m/qvtoml/downloads/drops/3.0.0/S200912160721/m2m-qvtoml-SDK-3.0.0M4.zip_


    > -URL
    > _
    __http://download.eclipse.org/modeling/mdt/ocl/downloads/drops/3.0.0/I201001070745/mdt-ocl-SDK-I201001070745.zip_


    >
    > Cheers...
    > Anthony
    > --
    > Anthony Hunter _mailto:anthonyh@xxxxxx.com_
    > Software Development Manager
    > IBM Rational Software: Aurora / Modeling Tools
    > Phone: 613-270-4613
    >
    > _______________________________________________
    > gmf-dev mailing list
    > _gmf-dev@eclipse.org_ <mailto:gmf-dev@xxxxxxxxxxx>
    > _https://dev.eclipse.org/mailman/listinfo/gmf-dev_
    >
    >
-- Nick Boldt :: JBoss by Red Hat 
    Productization Lead :: JBoss Tools & Dev Studio
    Release Engineer :: Dash Athena _
    __http://nick.divbyzero.com_ <http://nick.divbyzero.com/>
    _______________________________________________
    gmf-dev mailing list _
    __gmf-dev@eclipse.org_ <mailto:gmf-dev@xxxxxxxxxxx> _
    __https://dev.eclipse.org/mailman/listinfo/gmf-dev_






    _______________________________________________
    dash-dev mailing list
    _dash-dev@eclipse.org_ <mailto:dash-dev@xxxxxxxxxxx>
    _https://dev.eclipse.org/mailman/listinfo/dash-dev_

    _______________________________________________
    dash-dev mailing list
    dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
    https://dev.eclipse.org/mailman/listinfo/dash-dev


    _______________________________________________
    dash-dev mailing list
    dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
    https://dev.eclipse.org/mailman/listinfo/dash-dev





    _______________________________________________
    dash-dev mailing list
    dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
    https://dev.eclipse.org/mailman/listinfo/dash-dev
-- EclipseCon 2010*Denis Roy* 
    Manager, IT Infrastructure
    Eclipse Foundation, Inc. -- http://www.eclipse.org/
    Office: 613.224.9461 x224 (Eastern time)
    denis.roy@xxxxxxxxxxx <mailto:denis.roy@xxxxxxxxxxx>


    _______________________________________________
    dash-dev mailing list
    dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
    https://dev.eclipse.org/mailman/listinfo/dash-dev



    _______________________________________________
    dash-dev mailing list
    dash-dev@xxxxxxxxxxx <mailto:dash-dev@xxxxxxxxxxx>
    https://dev.eclipse.org/mailman/listinfo/dash-dev




--
Nick Boldt :: JBoss by Red Hat
Productization Lead :: JBoss Tools & Dev Studio
Release Engineer :: Dash Athena
http://nick.divbyzero.com
------------------------------------------------------------------------

_______________________________________________
dash-dev mailing list
dash-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/dash-dev

Back to the top