[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [ptp-dev] Questions related to batch job submission
|
Some more thoughts on 1). LSF seems to have something called a 'job
array' (http://hpc.ilri.cgiar.org/documents/admin_6.0/
G_jobarrays.html) which sounds kind of like what you're talking about
in that you can set up dependencies between jobs, etc. The whole
thing is submitted using a single submit command. It would seem to be
fairly straight forward to add an LSF specific launch configuration
page that lets you build up this array, then sends it as a bunch of
attributes on the submission command. One new job event would be sent
that is the 'array' job and returns the submission ID, then new job
events for each of the individual jobs in the array could also be
sent. The LSF proxy could then distinguish commands sent to the
'array' job ID or the individual job IDs.
Seems like something similar could work for LL.
Greg
On Jun 18, 2007, at 1:38 PM, Dave Wootton wrote:
Hi
We were discussing details of batch job submission thru PTP and had
some
questions about expected PTP behavior and how we should implement our
support
1) The user can submit a job which contains a set of job steps.
From our
perspective, each job step behaves as if it was a separate job,
although
there may be dependency and conditional execution specifications that
require job step 1 to complete before job step 2 canm begin, or
that job
step 2 can only run if job step 1 completed successfully, etc. The job
submission/job command file that specifes the individual job steps
is a
single file that will be passed to the proxy in a single run command.
Current ptp behavior is that the run command includes a jobid that is
generated by the front end and passed to the proxy. The proxy
responds to
the run command with an event containing that jobid as well as the
proxy-generated identifier for that job. This works for a single
job, or
for the first step of a multi-step job.
How should multi-step jobs be handled? Should the PTP front end have a
list or array of jobs steps built at the time the job is submitted,
and
use the same jobid for each of those steps? Should the front end
generate
a unique jobid for each step that is then passed across in the run
command, maybe as an array of jobids, and then the proxy generate a
new
job event for each step using the corresponding jobid? Should the
proxy
just use the passed jobid for the first step and use -1 as the
jobid for
all subsequent steps, since the front end doesn't know about the
additional job steps?
2) When we submit a job, the job may not appear on any job queue for a
while, possibly several minutes. We won't have some job related
information, such as cluster (machine name) where the job was queued,
until the job appears on the job queue. If we delay our event
response to
the run command until we have the required information, does that
cause
problems, such as blocking any additional jobs from being queued
until the
event notification from the first run command is received? Does the
front
end have problems tracking multiple 'in process' run commands
active at
the same time?
Thanks
Dave
_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/ptp-dev
