Greg
I think the proxy should be responsible for building the routing
file, in
order to keep the traffic on the connection between the GUI and the
proxy
down. With the current approach, you are sending node information
across
the connection twice, once to populate the PTP runtime model, then a
second time to create the routing file on the nodes where the SDMs are
running. I'm not sure what the message length for the messages from
the
proxy to the GUI are, but for the remote_file you have
strlen(task_index)
+ strlen(hostname) + strlen(port_number) + 3 bytes per node. In my
case
that's close to 20 bytes per task, minimum. With large numbers of
tasks,
this could be a lot of data, and since all of these interactions
between
the GUI, the proxy, and the SDMs are a serial process, they slow down
debugger startup.
The down side to this is the need for each proxy to implement
support for
each of unique debugger startup sequences it is willing to support,
where
you could end up with some proxies not supporting a debugger. If you
implement all of the code in the GUI resource manager side though,
I'm not
sure you don't have the same problem, where the RM needs to be aware
of
the details of both the debugger startup sequence and the details of a
particular runtime environment/proxy.
The other question I have after seeing the contents of the routing
file
you generate is the generation of random port numbers. If you end up
actually using these port numbers, do you run the risk of accidentally
using a port number reserved for some other application, unless you
block
out a range of port numbers and only use that range? Even if port
numbers
are up for grabs with no expectation of reserved port numbers, what
happens if something else is using your port number?
Dave
Greg Watson <g.watson@xxxxxxxxxxxx>
Sent by: ptp-dev-bounces@xxxxxxxxxxx
08/27/2008 06:48 PM
Please respond to
Parallel Tools Platform general developers <ptp-dev@xxxxxxxxxxx>
To
Parallel Tools Platform general developers <ptp-dev@xxxxxxxxxxx>
cc
Subject
Re: [ptp-dev] Questions about PTP SDM debugger
Dave,
It's not an SDM server generating the routing file, it's the java
side of
the SDM, then the routing file is transferred to the remote system.
We've
been debating where this code should live, so I'd welcome input from
you
on this.
Currently the way it works is like this:
1. User launches debug job.
2. Launch plugin calls debugger initialize()
3. Debugger allocates port number for debugger connection
4. Launch plugin calls submitJob command with debug flag to launch
the SDM
servers
5. When the SDM processes are all running, the launch plugin calls
createDebugSession()
5a. createDebugSession() generates the routing file from the runtime
model
and transfers it to the remote system
5b. createDebugSession() starts a thread which launches the SDM master
process
Now in my mind, steps 5a and 5b are really something that the resource
manager should be responsible for, since the proxy may be able to
generate
the file more efficiently, or may want to start the master in a
particular
place. However, this would then mean that the proxy will need to
know the
type of debugger that it is going to launch, since if we're going to
be
generic we need to support other debuggers, and each debugger may
have a
different launch sequence.
Anyway, if you have any thoughts or suggestions, they would be
appreciated. We can easily change (or disable) the code to work with
your
proxy.
Regards,
Greg
On Aug 26, 2008, at 10:52 PM, Dave Wootton wrote:
Greg
My proxy now starts both the top level SDM as well as the individual
process SDMs. I'm actually creating the individual process SDMs (via
'poe
sdm ...') before I create the top level SDM since that seemed like a
simpler way to fit the creation of these within my existing proxy
logic.
I'm not sure exactly which gets created first by the system since the
individual process SDMs are being invoked in the child process leg
following a fork() call in my proxy while the parent leg of the fork()
issues a second fork() and invokes the to level SDM on that fork()'s
child
leg. From what I understand so far about how the SDMs are supposed
to do
nothing until the routing_file is created, I don't think the timing
is a
problem.
I have a bug with my code that is supposed to be creating the
routing_file
that I need to track down.
However, the odd thing going on that I don't understand, is that one
of
the SDMs is creating the routing file on it's own. I get the following
messages that I think are coming from the top level SDM
debug: waiting for connect
PE@k17sf2p03 (RDT): effsize: 3, size: 2, rv: 0
PE@k17sf2p03 (RDT): nodeID: 1, hostname: k17sf2p03, port: 15411
PE@k17sf2p03 (RDT): nodeID: 0, hostname: k17sf2p03, port: 10459
PE@k17sf2p03 (RDT): SDM[2]: [1] No port found for the sdm child.
hostname:
k17sf2p03
PE@k17sf2p03 (RDT): SDM[1]: sdm_init failed
PE@k17sf2p03 (RDT): 08/26 22:33:54 T(256) Trace: +++ Pid 28627 exited
PACKET:[00000017PE@k17sf2p03 (RDT): 08/26 22:33:58 T(256) Trace: >>>
terminate_job entered. (Line 1412)
PE@k17sf2p03 (RDT): 08/26 22:33:58 T(256) Trace: 'jobId=4
PE@k17sf2p03 (RDT): '08/26 22:33:58 T(256) Trace: <<< terminate_job
exited. (Line 1438)
PE@k17sf2p03 (RDT): 08/26 22:33:58 T(256) Trace: >>> kill_process
entered.
(Line 3486)
0000:00000016:00000000]
PE@k17sf2p03 (RDT)ProxyRuntimeClient received event 0 transid=22
calling session finish
debug: received message event
accept thread exiting...
Msg: EventRequestManager - addEventRequest(): Request: Suspend
request in
status [UNKNOWN] for {0}.
Msg: EventRequestManager - addEventRequest(): Request: Terminate
request
in status [UNKNOWN] for {0}.
Msg: EventRequestManager - addEventRequest(): Request: Stop debugger
request in status [UNKNOWN] for {}.
**** Msg: AbstractEventManager - registerEventRequest(): Request:
Suspend
request in status [UNKNOWN] for {0}.
**** Msg: AbstractEventManager - notifyEventRequest(): Request:
Suspend
request in status [ERROR] for {0}.
Error completing debug job launch: Cannot connect to debugger
PACKET:[00000017PE@k17sf2p03 (RDT): 08/26 22:34:04 T(256) Trace: >>>
terminate_job entered. (Line 1412)
(RDT is the name for my proxy)
If I look in the current working directory for my proxy, I see a
'routing_file' which has the contents
2
1 k17sf2p03 15411
0 k17sf2p03 10459
This isn't anything my proxy is generating, so I'm not sure where it's
coming from.
Any ideas what's going on? Am I misunderstanding what's supposed to be
going on?
Dave
Greg Watson <g.watson@xxxxxxxxxxxx>
Sent by: ptp-dev-bounces@xxxxxxxxxxx
08/26/2008 08:35 AM
Please respond to
Parallel Tools Platform general developers <ptp-dev@xxxxxxxxxxx>
To
Parallel Tools Platform general developers <ptp-dev@xxxxxxxxxxx>
cc
Subject
Re: [ptp-dev] Questions about PTP SDM debugger
Dave,
I didn't write this code, but it sounds like 10 seconds is probably
too
short for the timeout as you say. It would probably be better to
have the
master sdm wait forever since it can be killed if the debug launch
needs
to be aborted.
I believe the race condition should be dealt with already. The first
line
of the file contains the number of entries, so the SDM will not
consider
the file complete until it contains this many routing entries. I
think it
just re-reads the file after some delay until the count is correct.
Greg
On Aug 25, 2008, at 10:46 PM, Dave Wootton wrote:
Greg
I got far enough with my experimentation that I can now get a top
level
SDM started and not exit. I may have had other problems, but once I
turned
on sdm debug I found that there's code in the sdm_tcpip_init
function that
loops for 10 seconds trying to find the routing file (which it looks
like
is named 'routing_file' in the current directory). If the file isn't
found
within 10 seconds, sdm issues a timeout message and exits. I changed
the
timeout to 1000 seconds and the sdm does not exit. So I think I have a
starting point to continue working on this.
From what I understand of the flow you explained, I don't think 10
seconds will be long enough even once I get my proxy to generate the
routing_file. As I understand it, I need to create the top SDM, then
start
the individual task SDMs by 'poe sdm ...', wait for poe to generate
the
attach.cfg file that gives me the mapping from application task rank
to
node and pid for each task, then create the routing_file using the
attach.cfg file as input, and then the debugger will take off. I think
this approach would work for the LoadLeveler case as well, since the
attach.cfg file still gets generated. However, on a slow system, or
for an
application with a large number of tasks, it could take several
minutes
for this processing to complete.
With a large enough number of tasks, the creation of the
routing_file may
not complete before the individual SDMs detect it and try to process
it.
What happens then? Is there logic in the SDM to retry reading the
routing_file until it gets a complete copy?
Dave
Greg Watson <g.watson@xxxxxxxxxxxx>
Sent by: ptp-dev-bounces@xxxxxxxxxxx
08/25/2008 04:18 PM
Please respond to
Parallel Tools Platform general developers <ptp-dev@xxxxxxxxxxx>
To
Parallel Tools Platform general developers <ptp-dev@xxxxxxxxxxx>
cc
Subject
Re: [ptp-dev] Questions about PTP SDM debugger
On Aug 25, 2008, at 11:00 AM, Dave Wootton wrote:
Greg
Some additional questions
1) It looks like I don't pass the name of the application executable
as a
parameter on the top level SDM instance since the top level instance
isn't
directly invoking the SDM instances required for individual tasks.
No this isn't necessary. The debugger protocol supplies the executable
name and the application arguments.
2) What are the invocation parameters of the individual SDM? I'm
sort of
guessing I need the hostname and port of the top SDM, the pathname
of the
application and any parameters the application requires. I'm
guessing then
the individual SDM starts, starts a debugger instance and the
debugger
instance starts the application instance.
The master sdm should be invoked with as 'sdm --host=address --
port=port --debugger=gdb-mi --numprocs=n' where address is the address
of the machine running eclipse and port is a port number assigned by
PTP. The servers will be started with something like 'mpirun sdm -
debugger=gdb-mi --numprocs=n'.
3) Is the routing file on a node a list of all tasks in the
application or
only the tasks running on that node?
A list of all tasks.
4) How does the routing file get loaded onto each individual node?
At the moment it is assumed there is a shared filesystem. This
requirement will be removed in a later version, and the sdm's
themselves will be used to propagate the routing file.
5) How does each individual SDM know how to connect back to the top
SDM if
the top SDM host/port is not a parameter?
Connections propagate up the tree (starting from the master). Each sdm
knows the index of its children (computed as a binomial tree) so it
just attempts to connect to its children using the address/port
obtained from the routing file.
6) If the individual SDM is passed the host/port that it connects to
the
top SDM, how do I find out what that top level SDM port is?
There is no easy way to do this at the moment, since it is generated
internally and passed to the submitJob command as an argument. The
easiest way would be to print out the arguments to the submitJob
command either in the Java side of the RM or in your proxy.
I think I understand how this is supposed to work, and it seems
reasonable
for the case where the user specifies a host list file. In the case
where
we use LoadLeveler to allocate nodes, I'm not sure how this will work
since we have no way of knowing what nodes are allocated until the
poe job
(the SDMs) starts.
The SDMs do nothing until they get the routing file. Would it be
possible to launch the SDMs, get the node information from LL, then
create the routing file? This is how the new OMPI RM works.
Greg
_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/ptp-dev
_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/ptp-dev
_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/ptp-dev
_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/ptp-dev
_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/ptp-dev
_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/ptp-dev
