Re: [equinox-dev] [prov] processing steps, restartable downloads and EC

[Jeff McAffer <Jeff_McAffer@xxxxxxxxxx>]

Scott Lewis wrote on 09/10/2007 12:58:10 PM:

> Actually, I'm a little surprised that you have so far passed the 
> ProcessingSteps as output streams directly to the ECF OutputStream,
> as I was expecting that you would have a temporary file to receive
> the file contents, and then when the file reception is done *then*
> apply the ProcessingSteps.  

Is there something wrong with passing the data along
immediate?  I guess it would be a trade off.  Some steps only
work on whole files while others can be streamed.  Streaming is more
effecient since you have the bytes right there.  there may be some
buffering (e.g., performance) but there are several opportunities for optimization.
 We could also add a step at the beginning of a chain that buffers
up the data in a file and then pumps it through.  This sort of step
could be used to do the restarting but if ECF already had (or could have)
something, that seems like more fun.

> But in any event, we can add impl support for
pause/resume/caching 
> etc to the ECF receive implementations w/o changing API to support
> required use cases.  I would appreciate a little better 
> understanding of the existing ProcessingSteps and their function...
> so could someone point me at the relevant packages/classes and I'll
> take more of a look?

The current code is in my workspace as I have been
doing some significant reworking and have not finished.  I will post
to this list when it is released.

> Seems like this would also be a good topic for
the upcoming Equinox 
> Summit:  what enhancements are needed for file transfer both
at API 
> and impl:  e.g. pause/resume enhancements, file caching, 
> monitoring/transfer statistics collection?, support more/other providers,
etc.

yes.  I added this to the list.  Under your
name :-)

Jeff