[stem-dev] Project Structure Change Proposals: Features, Update Site, B

[Matthew Davis <mattadav@xxxxxxxxxx>]

Hi All,

As has been discussed on our weekly phone call, upcoming additions to STEM 
are driving the need for changes in how we distribute project components. 
Notably, due to the addition of new Earth Science data, our binary 
distribution would more than double in size.  The following is a list of 
actions already taken and proposals for discussion on how to augment, 
extend, and support these changes.

The following proposals and changes will be applied as of STEM version 
1.2.1.  I appreciate any comments, feedback, or other discussion regarding 
these proposals. 

My apologies for the long email.

Thanks,
-Matt




P2 Repositories and Update Sites
---------------------------
To make retrieving and using this new data in STEM easy, we will begin 
distributing optional data using a P2 Repository/Update Site.  This will 
allow STEM users to retrieve/install automatically using the "Install New 
Software" menu in STEM.  For now, the only things we will distribute 
through the update site are static (generated) data features/plugins.  We 
will not distribute STEM application updates at this time, although this 
initial groundwork can be used to move to that in the future.

By enabling P2 in STEM, we also get additional functionality of the 
provisioning architecture, such as the "dropins" folder.  This may be 
useful in the future.

* Proposal For Update Site URL for Additional Data plug-ins:

http://download.eclipse.org/stem/update-site/addons/

Because the data is historical and should not change, the update site will 
be statically built and distributed for now.  We will not integrate it 
into the build process.




Feature-oriented Builds
---------------------------
As a prerequisite to using P2 Repositories and Update Sites, we have made 
initial changes to the STEM Product and build process to become 
"feature-oriented" (versus "plug-in-oriented").  The initial changes 
extend existing work that someone else began (Dan?).  The following is a 
proposal with discussion on how to proceed.

* Initial/Current Features

Before I began looking at this, the following features existed, but were 
unused, in the STEM repository:

org.eclipse.stem.feature - Top-level container that represents the STEM 
application.
org.eclipse.stem.feature.prereq - Third-party (non-STEM) dependencies, 
including Eclipse Platform, EMF, BIRT, Zest.
org.eclipse.stem.feature.core - The plug-ins that make up the "core" 
functionality of STEM

In addition to tweaking these features, I added a new feature using the 
existing naming scheme:
org.eclipse.stem.feature.internal.data - The plug-ins that contain raw 
data and generator code to create data distributed with STEM

* Naming Schemes

Dan pointed out a couple weeks ago that this naming scheme is inconsistent 
with the naming scheme used by other Eclipse projects.  I agree, and I 
propose that we change the naming scheme to conform.  I think there are 
two options:

1. Append ".feature" to plug-in grouping id
This appears to be the most widely used approach (and was suggested by 
Dan).  The features would be renamed to look like this:
org.eclipse.stem.feature
org.eclipse.stem.core.feature
org.eclipse.stem.internal.data.feature
etc

OR

2.  Remove "feature" altogether from the name
This is currently in use by the EMF project.  As plug-ins and features can 
share names, you can simply drop the .feature from the feature name (for 
workspace compatibility, however, you do need to append -feature to the 
directory).  In this case, the features would be renamed as such:
org.eclipse.stem (directory name org.eclipse.stem-feature)
org.eclipse.stem.core (directory name org.eclipse.stem.core-feature)
org.eclipse.stem.internal.data (directory name 
org.eclipse.stem.internal.data-feature)
etc

* How to split STEM into Features

This is the most important question - how exactly do we split STEM up into 
"features".  Remember that an Eclipse "feature" represent discrete but 
integrate-able groupings of Eclipse plug-ins and other features. 
Basically, a feature groups together plug-ins/features that provide 
specific new functionality and/or extends existing functionality in an 
Eclipse-based application.

With that in mind, we re-open an old, mostly philosophical question --- 
"What is STEM?"

By design, STEM is an event modeler that can model and simulate anything 
that can be represented on a graph.  The core of STEM was designed 
specifically with this in mind.
In practice, STEM is an epidemiological modeler that models and simulates 
infectious diseases.

After looking at the relationships between components, there is no 
absolute separation between generic modeling components and 
epidemiological-oriented components.  As a result, it's currently not 
possible to create a clean split between generic and disease-specific 
functionality.  What I mean by this is:  there's no way to create isolated 
"core" and "epidemiology" features.  That said, there are "epidemiology" 
components that can be split off into their own groupings.

I propose the following features be created in STEM.  Features with a (*) 
will be part of the binary distribution on the Web site.  Other features 
will be distributed via Update SIte.  The actual breakdown of plug-ins for 
each feature will be determined after discussion.

* Proposed Features

(*) org.eclipse.stem.feature - Top level feature, target of the builder. 
Contains references to other (*) features.
 + (*) org.eclipse.stem.prereq.feature - Third-party dependencies
 + (*) org.eclipse.stem.core.feature - Core functionality, including core, 
definitions, basic geography, and core UI.  Effectively the minimum set of 
plug-ins required to run STEM
 + (*) org.eclipse.stem.epidemiology.feature - Contains plug-ins 
pertaining to epidemiology that are  NOT required by default to run STEM. 
Mostly the disease models and their UI plug-ins.
 + (*) org.eclipse.stem.data.geography.feature - Political geography data, 
including country-level maps, names, and associated graphs and models
 + (*) org.eclipse.stem.data.population.human.feature - Human population 
data plug-ins, including associated graphs and models
 + (*) org.eclipse.stem.data.transportation.feature - Transportation data 
plug-ins, including associated graphs and models

org.eclipse.stem.data.earthscience.feature - Earth Science Data plug-ins
 + org.eclipse.stem.data.earthscience.[2000-2009].feature - Individual 
years of earth science data.

org.eclipse.stem.tests.feature


Source Code Repository Restructure
---------------------------

1. Do we want to restructure the repository?

As of this morning, there are 147 plugins, features, and other projects in 
the STEM source root.  That is a lot.  And, as is the case with our 
changes to enable extra functionality to be distributed outside of the 
main STEM application, there's really no need to check out all these 
projects.

Other Eclipse projects have more structure in their repositories, 
separating various components.  Adding more structure, of course, makes 
the checkout process more complicated ("What do I need, and where does it 
live?").

2. If yes, then when?

A repository restructure is a complicated affair, where the easiest way to 
approach it from a developers point of view is to commit all diffs, freeze 
changes, clean you workspace, and re-checkout.  We have discussed the most 
logical time would be when we switch from Helios to Indigo, probably in 
July.

3. What does the new structure look like?

This is a tough question.  There's not a lot of consistency across Eclipse 
projects for this, so we're really open.  I'm going to take a stab at a 
proposal that incorporates ideas from other Eclipse projects.

/trunk/
/trunk/features - Contains the 'feature' projects
/trunk/core - Contains the 'core' projects, that make up the 'core' 
feature
/trunk/epidemiology - Contains the epidemiology-centric projects (make up 
the 'epidemiology' feature)
/trunk/data/skeletons - Contains the data plug-in skeletons
/trunk/data/internal - Contains plug-ins that are intended for internal 
use only (internal.data plugins)
/trunk/tests - Contains the test case projects
/trunk/releng - Contains projects related to release engineering
/tags/*
/branches/*


An alternate suggestion is as follows:

/stem/trunk/features
/stem/trunk/core
/stem/trunk/epidemiology
/stem/trunk/tests
/stem/trunk/releng
/stem/tags/*
/stem/branches/*

/data/trunk/features
/data/trunk/internal
/data/trunk/skeletons
/data/trunk/tests
/data/tags/*
/data/branches/*


We can then distribute a project set file (PSF) that contains the 
appropriate references to automatically checkout and build a developer's 
workspace depending on what they want.  The Eclipse Metadata contains 
provisions for this.

4. Subversion vs. Git

A lot of Eclipse projects are moving to Git for their source repository 
over CVS/SVN.  If we want to do this in the next year or two, this will be 
the time to do it.



Changes to Build Process
---------------------------
* Java 6 Base Requirement
As was voted on several weeks ago, we are now using a Java 6 JDK with 1.6 
source and class compatibility requirements in our builder.  This 
effectively deprecates Java 5.

* Feature-oriented PDE Build, P2 enablement 
Additional changes to the build process to support feature-oriented builds 
and enable P2 repositories have been made.  These changes are committed 
into the org.eclipse.stem.releng.feature project, but will be rolled into 
the org.eclipse.stem.releng project after the final changes are in place

* Use of Hudson/Jenkins for automated builds
We've set up a Jenkins instance on an internal server to handle running 
and automating STEM builds.  I realize that Eclipse provides projects with 
infrastructure to do this in the open, but we have significant hardware 
assets and there's really no need to place additional burdens on their 
infrastructure.  I will publish the Jenkins/Hudson configuration files for 
anyone that wants to set up their own automated STEM builds.

* Our plan right now is to run nightly builds to test code compliance but 
we will not externally publish these builds (as we're talking 2 GB total 
per build).  Weekly, probably on Sunday morning, we will run a build that 
will be published automatically to the STEM Web site.

(Side question of whether we should use Hudson or Jenkins:  We will 
standardize on whatever Eclipse standardizes on.  If the Eclipse Hudson 
project is approved, and they resolve all IP concerns, we will use Hudson 
upon the first Eclipse/EPL release.  Until then, we will use Jenkins).

* Automatic updating of download site
I've made a handful of changes to the PHP script that controls downloads 
to enable automatic updating of build versions.  These changes will be 
documented in the wiki.

* Use of Buckminster and/or Target Platforms
Currently out of scope for this discussion.  If someone wants to start a 
side conversation - and implement - the build process using Buckminster, I 
encourage it.  However, this was not a part of this project.