Dave,
I agree that stdout is one area that needs a lot of work from the scaling perspective. I wonder if PProcess is actually the best place to handle stdout though? Maybe this might be better handled by the UI views that deal with the output (ParallelProcessView and JobConsole). It seems more like management of the text control buffers, rather than something independent of the UI.
Also, at the moment stdout/stderr is handled as events sent to the runtime model. This is a nice simple approach, since it avoids the need to create extra sockets, etc., but also makes it very easy to overload the event processing framework. Since other activities can also do this, it would be nice to have some kind of throttling or flow control mechanism.
Cheers, Greg On Dec 14, 2009, at 4:14 PM, Dave Wootton wrote:
Greg
I'm starting to look at some of the
large system scaling problems in PTP. It looks like a fairly simple place
to start working is the code which handles stdout processing for tasks
in the application. Currently, it looks like an output file is generated
for each application task when the corresponding PProcess object is created.
The result is that the directory specified by the PTP preferences output
directory setting gets filled with a lot of 0-byte files, unless each application
task actually writes to stdout. This results in several problems
1) An inode (or windows equivalent)
of disk space is consumed for each application task
2) System time is spent creating files
that may never be used. With a large number of tasks, the overhead is probably
substantial
3) PTP is holding file resources for
each task, which could consume all available file handles. I'm not sure
what the file handle limit in windows is, but suspect it's probably small.
4) The output directory has a large
number of files in it, which could result in poor performance when listing
(or browsing via file selector dialog), etc that directory
I think the place to start is to have
the PProcess constructor set up the filename to be used for the task, but
to not create a new OutputTextFile object. Then PProcess.addOutput() is
modified so that it checks for the existence of the OutputTextFile and
creates it if needed (on the first call to that method). Methods which
access outputFile for viewing, cleanup, etc) need to be modified so they
handle the case of a non existent file without raising exceptions or other
errors. I think that if the user double clicks an icon in the jobs view
for a task which has no stdout output, that he gets a popup stating there
is no stdout for that task. Alternatively, should the icon present some
state (maybe disabled) indicating there is no output (that could result
in additional state processing as the user scrolls thru the set of icons
in the job view, and therefore may not be a good idea)
Does this make sense?
Dave_______________________________________________ ptp-dev mailing list ptp-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/ptp-dev
|