[equinox-dev] [prov] [ecf-dev] E-intro [Was Efficient downloads]

[Jeff McAffer <Jeff_McAffer@xxxxxxxxxx>]

This thread in the ECF list is interesting.
 We already enable the use of ECF as a transport layer so we should
be able to use this new proposed capability and robustness.
Jeff

----- Forwarded by Jeff
McAffer/Ottawa/IBM on 06/02/2007 09:15 PM -----

Filip Hrbek <filip.hrbek@xxxxxxxxxxxxxx>
Sent by: ecf-dev-bounces@xxxxxxxxxxx

05/30/2007 11:38 AM

Please respond to
"Eclipse Communication Framework \(ECF\) developer mailing list."        <ecf-dev@xxxxxxxxxxx>

To	"Eclipse Communication Framework (ECF) developer mailing list." <ecf-dev@xxxxxxxxxxx>
cc
Subject	Re: [ecf-dev] E-intro [Was Efficient downloads]

Hi Scott,

thanks for your quick reply.

We need a very robust file transfer implementation for our application.
I believe that if we cooperate on this we'll create something really good.

Purpose:
- have downloads as fast as possible (optimize downloads for case that
the download capacity is bigger than single location's upload capacity)
- have downloads reliable (download everything or nothing, avoid saving
corrupted files if possible)
- have downloads convenient (automatic recovery, discovery of alternatives)
- let downloads have good UI (verbose enough, good cancel/pause/resume
response)

Let me list features we are looking for (just a brainstorming, no 
specification yet):
- multiple protocol support (http/https/ftp/file...)
- pause/resume using partial content requests (connection can be 
closed/reopened before an attempt to resume)
- resume from a different location (e.g. different mirror)
- multi threaded download of one artifact from one location (starting at
different content offsets)
- multi threaded download of one artifact from several locations at the
same time
- multi threaded download of several artifacts at the same time (using
either single threaded or multi threaded single artifact downloads)
- retrieving information from special headers (like Content-Disposition)
- detecting URL redirections to final mirrors
- detecting approximate user's geographical position
- collecting download statistics (slow/unreliable mirror discovery)
- dynamic switching between mirrors in case of slow download
- automatic recovery after connection timeout
- good API for verbose progress bars (ability to create a good UI)
- good API for statistics processing
- configurable behavior (enable configuring features described above)
- ...and perhaps many more (?)

I don't know how much from this you already cover. I'll study what you
currently provide first. All the features don't have to be implemented
at the same time, we can define development stages to start with more 
important issues and continue with less important ones.

Best regards
  Filip Hrbek

Note: I'll be offline from this Friday until next Sunday, so don't worry
if I don't answer your emails during next week. I'll be definitely back
from Monday (11th June).



Scott Lewis napsal(a):
> Hi Thomas and Filip,
>
> Thanks!  The help is most welcome.
> The filetransfer API is in plugin:  org.eclipse.ecf.filetransfer.
> The bundle docs (with pointers to javadocs) is:  
> http://wiki.eclipse.org/index.php/ECF_API_Docs#File_Transfer_API
>
> There are also pointers in wiki to project sets source and test code.
>
> There is a base provider plugin:  
> org.eclipse.ecf.provider.filetransfer.  This plugin has an extension
> point that allows other plugins to insert implementations associated
> with various protocols.  See the extension point 
> org.eclipse.ecf.provider.filetransfer.fileTransferProtocolFactory.
  
> The httpclient implementation uses this extension point (and is 
> defined in another plugin:  org.eclipse.ecf.provider.filetransfer.http).
>
> The basic implementation that uses jobs for doing pause/resume 
> (IFileTransferPausable) is in 
> org.eclipse.ecf.provider.filetransfer.retrieve.AbstractRetrieveFileTransfer.
 
> This can/could be used (and appropriate methods overidden) or 
> replaced...up to you as to what makes sense.
>
> Filip please let me and everyone know if you have any 
> questions/comments/suggested or needed changes...and thanksinadvance!
 
> I would/will be curious myself about what http/https impl you are
> looking to use and what specific http/https protocol features you
will 
> be using as well.
>
> Scott
>
> Thomas Hallgren wrote:
>> Hi Scott,
>> We, (the Buckminster team) would like to offer our help in 
>> implementing protocol-level pause/resume and recovery capabilities
>> for the ECF http/https file transfer. I'd like you to meet Filip
>> Hrbek. He is the Buckminster guy who currently works on improving
our 
>> materialization. He will be our main resource for this, should
you 
>> accept our help. So Filip, meet Scott. Scott meet Filip.
>>
>> Regards,
>> Thomas Hallgren
>>
>>
>>
>> Scott Lewis wrote:
>>> Hi Thomas,
>>>
>>> Thomas Hallgren wrote:
>>>> Hi Scott,
>>>> Thanks for a very elaborate answer. This looks very promising.
Well 
>>>> structured, low overhead. Only thing missing if I read
you 
>>>> correctly is true pause/resume on protocol level (and
the failure 
>>>> recovery that I think will come as a bonus). Do you have
any sense 
>>>> what it would take to implement that?
>>>
>>> It depends upon each protocol and it's corresponding support
in 
>>> (some) protocol library.  I assume you are most interested
in 
>>> http/https.  I'll take a look at the httpclient 3.0.1
support for 
>>> pause/resume and failure recovery.
>>>
>>> Scott
>>>
>>>
>>> _______________________________________________
>>> ecf-dev mailing list
>>> ecf-dev@xxxxxxxxxxx
>>> https://dev.eclipse.org/mailman/listinfo/ecf-dev
>>
>> _______________________________________________
>> ecf-dev mailing list
>> ecf-dev@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/ecf-dev
>
> _______________________________________________
> ecf-dev mailing list
> ecf-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/ecf-dev
_______________________________________________
ecf-dev mailing list
ecf-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/ecf-dev