[cross-project-issues-dev] p2 repositories ... we can do better

[David M Williams <david_williams@xxxxxxxxxx>]

That is, we can do better at educating people how to make a good p2
repository! We have not done that education well. Hopefully this note, with
its links, can help provide the education you need .... then tell your
friends! ... tell your family! :)

This information actually applies to all projects and all releases at
Eclipse, not just the yearly train, but I am sending to cross-project list,
thinking that will educate most projects as a first step. This should be
considered a requirement of correct and effective use of the Eclipse
Foundation Infrastructure so mentors and PMCs should make sure their
projects know about it and making effective use of the infrastructure.

I think everyone knows about pack200, and while sometimes a pain, it can
help reduce bandwidth for faster installs, and I think most projects do a
good job of using that. And that is one of the hardest things to do! ...
which makes me think the current situation is more about education than
amount of effort.

There are two things which, as a whole, we are not doing well.

A. Use p2.mirrorsURL property in artifact repositories (artifacts.jar/xml
files).

See bug 368826. There are many reasons for not using it ... such as lack of
tools, but another reason is that some of us (e.g. me) thought it was
"common knowledge" that you should use it, but of course it is not to
projects that are new to Eclipse. In response, I have added a little
documentation to the main IT Doc [2], beefed up and corrected some of the
community maintained page that describes how to use p2.mirrorsURL [3] and
provided some documentation on how to use a custom Eclipse Application to
set, change, or remove the p2.mirrorsURL, courtesy of the Webtools Releng
project [4]. While an "unofficial" tool, I think most can take advantage of
it, as is. Just practice on some test repos first! :) But this type of tool
can make it easier to fix existing artifact repositories without
hand-editing a 2 Megabyte XML file. But again, practice on some test repos
first! You can make a copy of (only) the artifacts.jar file to some /temp
directory your local machine and "pretend" that is the artifact repository,
to see if it makes the changes as you intend. Then use your scripts on
production machine, being sure to copy original artifacts jar to a safe
location first in case you need to revert, and be sure to test its really
working, afterwards, as is well described on the p2.mirrorsURL wiki[3].

B. Use p2.index at repository locations.

p2 likes metafiles for its metafiles!  As if content, artifacts,
composites, jars and xml were not enough, a while back p2 found the need to
use another file, called p2.index to help it figure out a few ambiguous
cases. Since few people need this functionality, it has gone ignored for a
long time but it is finally becoming noticeable, as Eclipse gets larger and
more popular (and, as a victim of its own success, as "p2 update" is
working well enough many people use that now, instead of downloading a
fresh zip file!). The "lucky accident" is that providing the p2.index file
can save p2 a few needless "round trips" to the download server, while it
figures out what is there, and what it should use, even if not truly needed
to resolve ambiguity. The long gory history is documented in bug 347448
[5]. In response, I have added a little documentation to the main IT Doc
[6] and beefed up and corrected the p2.index wiki [7]. The good news is
that it is easy to do. For 99% of you (I am guessing) it is simply a matter
of copying one of two files to your repository locations ... you do need to
pick which goes where (and most of you will need both, one at composite
sites, the other at simple sites). The two to pick from (for these
frequent, common cases) are documented on the wiki [7] (if you find other
common, or, even tricky cases, please add them to the wiki).  This is
especially important to have for "composite sites" since p2 normally checks
for that last, but if the p2.index says that location is a composite site,
then that saves p2 several needless "peek attempts" at download server.
Note, recent versions of b3 aggregator [8] produce these p2.index files
automatically, though the default Equinox p2 publisher does not (bug 302909
[9]) (I do not know about other p2 publishers, if there are any).

While all these issues and p2 in general can be made better, faster,
easier, and some day even mow your lawn (over time, if someone commits to
it) we are where we are, right now. So these are two things we
can/should/need to do right now to improve the situation: add p2.index
files to repository directories, and add p2.mirrorsURL property to artifact
repositories.

I know, this is a long note. And even tons more information linked to go
off and read for details. Keep this note for reference; bookmark the links.
Discuss it with your team. I suspect only one person per project needs to
understand it enough to act one it, but the main point is that there are
things that can be done right now (over the new few weeks, month) to
improve our p2 repositories (even old, existing ones -- the stats say even
helios repositories are still getting LOTS of use!) and then we would of
course clearly want to continue to do these good practices for Juno and
Kepler. Maybe one person per project could spend one day on it within the
next 2 to 4 weeks? (And, yes, I know that is asking for a lot of total
effort ... but, you will learn a lot of cool things in the process! ... and
cleaning up and learning these things now will help us be ready for the
Juno release!

Doing these things will give our Eclipse users a little better update and
install experience.  To be clear and honest, this is not gong to be like a
50% improvement, or anything ... I suspect it won't be noticeable most of
the time for most users ... but, some users will notice it a lot! if they
can get mirrors closer to their corner of the world (via p2.mirrorsURL
property), and it will be noticeable by many people during those "peak
times" then download.eclipse.org gets flooded by huge numbers of tiny
requests for files that will never exist (such as "peeking" for
content.jar, when p2 could look more directly for compositeContent.jar, if
p2.index was there).  [If our webmasters didn't know about p2, they'd
probably think it as a DOS attack :) just kidding, I've no idea what a DOS
attack looks like, and am pretty sure our IT infrastructure is responding
well.]

Three years ago, who would have thought p2 would be a victim of its own
success?! :)  I am sure it can be even more successful and more "enterprise
ready", so if anyone is interested, get involved! provide some patches!
make some tools!

As always, feel free to ask if questions ... and feel free to correct any
mistakes I make in my statements or my wiki updates.

Much thanks for your time and efforts.


p2.mirrorsURL related:

[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=368826
[2]
http://wiki.eclipse.org/IT_Infrastructure_Doc#Enable_mirrors_.2F_use_mirrorsURL_for_my_p2_repo.3F
[3] http://wiki.eclipse.org/Equinox/p2/p2.mirrorsURL
[4] http://wiki.eclipse.org/WTP/Releng/Tools/addRepoProperties

p2.index related:

[5] https://bugs.eclipse.org/bugs/show_bug.cgi?id=347448
[6]
http://wiki.eclipse.org/IT_Infrastructure_Doc#Include_a_p2.index_file_at_p2_repository_site.3F
[7] http://wiki.eclipse.org/Equinox/p2/p2_index
[8] http://wiki.eclipse.org/Eclipse_b3/aggregator/manual
[9] https://bugs.eclipse.org/bugs/show_bug.cgi?id=302909