Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [epp-dev] Files for release mirrors to have time

Thank you Denis.

Ed, do things look right now?

Jonah
~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Wed, 18 Mar 2020 at 09:31, Denis Roy <denis.roy@xxxxxxxxxxxxxxxxxxxxxx> wrote:

I've just cleared all the caches.


Denis

On 2020-03-18 9:26 a.m., Jonah Graham wrote:
:-(

I am looking at this now to see if I can understand. Denis/Webmaster if you can help that would be great as I don't understand how different clients are getting different responses.

Jonah



~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Wed, 18 Mar 2020 at 09:16, Ed Merks <ed.merks@xxxxxxxxx> wrote:

Jonah,

The problem from https://bugs.eclipse.org/bugs/show_bug.cgi?id=548497#c29 is happening again:

$curl -O http://download.eclipse.org/technology/epp/packages/2020-03/compositeContent.jar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   429  100   429    0     0   1618      0 --:--:-- --:--:-- --:--:--  1618

$curl -I http://download.eclipse.org/technology/epp/packages/2020-03/compositeContent.jar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   429    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/1.1 200 OK
Server: nginx
Date: Wed, 18 Mar 2020 13:06:13 GMT
Content-Type: application/x-java-archive
Content-Length: 429
Connection: keep-alive
Last-Modified: Fri, 06 Mar 2020 14:22:22 GMT
ETag: "1ad-5a03061c70fb5"
X-NodeID: download2
X-Proxy-Cache: STALE
Accept-Ranges: bytes

When I do this it bypasses the cache:

$curl -I --header "X-Cache-Bypass: true" http://download.eclipse.org/technology/epp/packages/2020-03/compositeContent.jar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/1.1 404 Not Found
Server: nginx
Date: Wed, 18 Mar 2020 13:09:15 GMT
Content-Type: text/html
Connection: keep-alive
Vary: Accept-Encoding
X-NodeID: download2

But a subsequent call still returns it as existing so it does not flush the cache:

$curl -I http://download.eclipse.org/technology/epp/packages/2020-03/compositeContent.jar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   429    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/1.1 200 OK
Server: nginx
Date: Wed, 18 Mar 2020 13:09:21 GMT
Content-Type: application/x-java-archive
Content-Length: 429
Connection: keep-alive
Last-Modified: Fri, 06 Mar 2020 14:22:22 GMT
ETag: "1ad-5a03061c70fb5"
X-NodeID: download2
X-Proxy-Cache: STALE
Accept-Ranges: bytes

Both compositeContent.jar and compositeArtifacts.jar need to yield 404 for a HEAD request...

Note how it says "X-Proxy-Cache: STALE" in each case.

Several calls like this:

$curl -I http://download.eclipse.org/technology/epp/packages/2020-03/content.jar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0 77058    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/1.1 200 OK
Server: nginx
Date: Wed, 18 Mar 2020 13:15:15 GMT
Content-Type: application/x-java-archive
Content-Length: 77058
Connection: keep-alive
Last-Modified: Tue, 17 Mar 2020 11:19:04 GMT
ETag: "12d02-5a10b1a79ee91"
X-NodeID: download2
X-Proxy-Cache: HIT
Accept-Ranges: bytes

Show "X-Proxy-Cache: HIT".

I think both those files were deleted two hours ago.

Regards,
Ed


On 18.03.2020 13:22, Jonah Graham wrote:
I believe everything is now in place for the release except that I and CI user cannot touch the directory so that mirrors are working for p2. See Bug 561215

Thanks,
Jonah

~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com


On Tue, 17 Mar 2020 at 09:55, Denis Roy <denis.roy@xxxxxxxxxxxxxxxxxxxxxx> wrote:

For completeness... Ed's assessment of file timestamps vs. directory is spot on, since p2 typically uses the directory as the mirrorsURL parameter.


Denis

On 2020-03-17 1:20 a.m., Ed Merks wrote:

Markus,

I believe that the server's general approach via download.php  is to consider the time stamp of the local file versus its timestamp on the mirror.  If they match, the mirror is a good one and will be returned in the list.  For the download folder, I expect the timestamp is not so important because only individual files will be downloaded from it. E.g., for this link, the timestamp of the file is important, but not the timestamps of any of the parent folders:

https://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/2019-12/R/eclipse-jee-2019-12-R-win32-x86_64.zip

Of course that's just a "theory"...

But for the repository itself, this URL is used by p2:

https://www.eclipse.org/downloads/download.php?format=xml&file=/technology/epp/packages/2020-03&countryCode=us&timeZone=1&format=xml

And here the timestamp of the 2020-03 folder itself is definitely important.

That all being said, an install will only need to download 4 artifacts from the EPP repo, while needing to download 100s from the train repo, so if there were no mirrors for the EPP repo, it would not cause such a huge problem as it does when there are no mirrors for the train repo. That repo already exists and has many mirrors:

https://www.eclipse.org/downloads/download.php?format=xml&file=/releases/2020-03/202003181000&countryCode=us&timeZone=1&format=xml

In less than 24 hours, this also appears to be fully mirrored:

https://www.eclipse.org/downloads/download.php?file=/oomph/epp/2020-03/R/eclipse-inst-win64.exe&countryCode=us&timeZone=1&format=xml

Regards,
Ed

On 16.03.2020 21:31, Markus Knauer wrote:
You are right. I used to be very careful with the modification times of files and directories, especially the download directories are important. I got used to using my TIMESTAMP files that can be found here and there on the download server, usually in commands like

  touch -r . TIMESTAMP
  touch -r TIMESTAMP .

to "store" the modification times of the current working directory, and to use it afterwards to copy the very same modification time back to it.

Regards,
Markus

On Mon, 16 Mar 2020 at 17:23, Ed Merks <ed.merks@xxxxxxxxx> wrote:

For last year's March release, the installer was downloaded almost 3 million times, and the packages themselves in total more than 1 million times. 

  https://www.eclipse.org/downloads/packages/release/2019-03/r

Of course it's not clear over which time period that happened, but there's always quite a bit of activity the day of and the days after...  The mirror failures right after the most 2019-12 release certainly brought that download server to its knees enough to take note that there was an ongoing problem.  And that's in December, close to Christmas time, when you wouldn't expect quite so much either.

That's all to say that I very much appreciate Jonah being conservative on this front.

Note too that I see that the mirrors are back now!

  https://www.eclipse.org/downloads/download.php?format=xml&file=/releases/2020-03/202003181000&countryCode=us&timeZone=1&format=xml
  https://www.eclipse.org/downloads/download.php?format=xml&file=/technology/epp/packages/2020-03/RC2a&countryCode=us&timeZone=1&format=xml

There are not so many mirrors for this yet:

  https://www.eclipse.org/downloads/download.php?file=/oomph/epp/2020-03/R/eclipse-inst-win64.exe

But that's after only a few hours.

Now that I'm thinking about it though, we should be careful what happens to the timestamp of the EPP folder and the train folder when the content/artifacts change just minutes before the release.  If the folder's timestamp changes, I believe all the mirrors will be disabled again for a while...

Regards,
Ed

On 16.03.2020 16:42, Markus Knauer wrote:
Full agreement... 24h is typically enough.
Sometimes, if it was really urgent in the past, I double-checked with Denis and the webmasters in order to make them aware that there may be a download peak. On the other hand, the peaks are not that high (especially for a March release) than they used to be.

And yes, I would definitely vote for the RC2a version (without aeri).

Regards,
Markus

On Mon, 16 Mar 2020 at 16:07, Jonah Graham <jonah@xxxxxxxxxxxxxxxx> wrote:


On Mon., Mar. 16, 2020, 10:54 Ed Merks, <ed.merks@xxxxxxxxx> wrote:

Jonah,

I believe 24 hours is normally more than enough, though the problem in https://bugs.eclipse.org/bugs/show_bug.cgi?id=561125 is still a problem so sooner is probably better than later.   I feel that it's safe to assume that RC2a versus RC2 will be good because the difference is really very trivial and I've confirmed that the Java, JEE and RCP packages installed from this basis via the installer start up fine...


Thanks Ed for confirming. I will do the downloads this evening (Ottawa time) 

Note that if you're going to transform the EPP repo into a simple repository, you should get the binary/features/plugins  folders copied up into the 2020-03 folder in preparation as well.

I will place all the binary/features/plugins files in place too, leaving just the artifacts and content files to be updated Wed morning. 

I will ensure that the composites are retuning 404 in time for the release too.

Thanks, 
Jonah 

Thanks for all the heavy lifting!!

Regards,
Ed


On 16.03.2020 15:42, Jonah Graham wrote:
Hi folks, 

When do the files need to be in place for release day? I assume today is better than tomorrow. 

In that case we need to have RC2a vs RC2 decision ASAP. If I don't hear anything further I will be making the RC2a files the R files. 

Thanks, 
Jonah 


_______________________________________________
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev
_______________________________________________
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev
_______________________________________________
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev



_______________________________________________
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev
_______________________________________________
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev



_______________________________________________
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev

_______________________________________________
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev
--

Denis Roy

Director, IT Services | Eclipse Foundation, Inc.

Eclipse Foundation: The Platform for Open Innovation and Collaboration

Twitter: @droy_eclipse

_______________________________________________
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev

_______________________________________________
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev
_______________________________________________
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev

_______________________________________________
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev
--

Denis Roy

Director, IT Services | Eclipse Foundation, Inc.

Eclipse Foundation: The Platform for Open Innovation and Collaboration

Twitter: @droy_eclipse

_______________________________________________
epp-dev mailing list
epp-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/epp-dev

Back to the top