Notes on Workspace Structure

In large Java development efforts, it is not uncommon for several inter-related projects to be under development at the same time. For the purposes of this discussion, assume teams, projects, and components align. All teams are working in the same repository, sharing a set of projects via a single stream; each project contains the source code for a single component.

Our example scenario has 4 projects: P1, P2, P3, and P4, with components C1 through C4, respectively. C1 is the only component that does not depend on any others; C2 depends on C1; C3 depends on C1 and C2; C4 depends on C2 (N.B., but not on C1 or C3).

We assume that there is a single CVS repository that contains all 4 projects in source code form. In addition, we assume that centralized builds are done periodically and posted to a web server where they can be downloaded as a unit. These downloads take the form of a zipped directory which includes a binary jar for each component, along with corresponding source jars to aid debugging.

We assume that each developer can download builds from the web server and install them on their local machine. To do their work, they set up Eclipse workspaces on their local machine and load one or more projects from the CVS repository. They also upgrade their workspace from time to time as new builds become available.

These assumptions are a plausible abstraction of what goes on in open source projects. (It is an open question as to how closely the Eclipse project with follow this work model.)

Developing, Using, and Patching Components

The following things are the norm for someone actively developing a component:

• source code is available for entire component
• source code is being browsed and changed
• source code is being compiled regularly
• source code changes are shared occasionally with other team members

• component is available as pre-compiled binary library
• source code is available for browsing but not for editing
• since used component is static, no changes to share with other team members

• component is available as pre-compiled binary library
• source code is selectively available for editing (patching)
• patched source code is compiled and used instead of pre-compiled binary
• patched source code is for local consumption (not shared with other team members)

The various scenarios presented below are all plausible workspace setups, each with certain advantages and disadvantages:

• Component plus libraries
• Source for everything
• Classpath variables to switch between libraries and sources
• Unshared proxy library projects
• Shared proxy library projects
• Stub projects
• Assorted Other Scenarios

Component plus libraries

• Team member's workspace contains just the project for their component.
• Each team members downloads latest binary build to c:\temp\build\ or some such and sets classpath variable BUILD = c:\temp\build\
• Build classpath references individual libraries relative to BUILD classpath variable.
• Use of classpath variable allows each developer to choose where to install the build.
• Classpath includes libraries for all prerequisite components.
• Pro: simple workspace setup (load a single shared project from repository and set single classpath variable).
• Pro: no restiction on number of jars per project.
• Pro: easy to upgrade to another binary build (unzip build in place, and close and reopen project in workspace to force a refresh).
• Con: unable to view code for a component that is not a prerequisite (requires loading an additional project).
• Con: unable to browse code for prerequiste components in proper content (code completion).
• Con: switching to developing another component involves loading another project and changing classpaths on dependent components (bad, because these are shared).
• Con: patching only via switching to active development.
• Note: uses classpath variables whose value is a library jar (works in R1.0).
• Note: this is roughly the way PDE 1.0 works.

Source for everything

• Team member's workspace contains all four projects and works entirely from source code.
• Build classpath references individual dependent projects by name.
• Classpath includes projects for all prerequisite components.
• Pro: able to browse code for all components.
• Pro: able to develop any component.
• Pro: easy to notice downstream impact of any changes made since all components are present.
• Pro: easy to keep current (by catching up with stream in repository).
• Con: Very slow workspace startup and large footprint because entails recompiling source code for all components.
• Con: Initial workspace setup involving multiple shared projects (automation possible).
• Con: difficulty handling projects with precompiled library jars that must be exported but do not exist in source (e.g., the Eclipse debugger's JDI jar)
• Note: uses required projects (works in R1.0).

project P1 (shared)
    build classpath = source /P1/C1
    output folder /P1/bin
&
project P2 (shared)
    build classpath = source /P2/C2; project P1
    output folder /P2/bin
&
project P3 (shared)
    build classpath = source /P3/C3; project P1; project P2
    output folder /P3/bin
&
project P4 (shared)
    build classpath = source /P4/C4; project P2
    output folder /P4/bin

Classpath variables to switch between libraries and sources

• Team member's workspace contains just the project for their component.
• Each team members downloads latest binary build to c:\temp\build\ or some such and sets family of classpath variables, one per component:
• P1_LIB = c:\temp\build\c1.jar
• P2_LIB = c:\temp\build\c2.jar
• P3_LIB = c:\temp\build\c3.jar
• P4_LIB = c:\temp\build\c4.jar
• Build classpath references an individual component via a classpath variable.
• Use of classpath variable allows each developer to choose where to install the build.
• Classpath includes entries for all prerequisite components.
• Pro: switch to development involves loading an additional project into workspace and rebinding a single classpath variable to refer to that project. For example, load project P2 and rebind P2_LIB = /P2.
• Pro: easy to upgrade to another binary build (unzip build in place, and close and reopen project in workspace to force a refresh).
• Con: difficuly attaching debug source to library jar
• Con: only works if there is exactly one library jar per project.
• Con: initial workspace setup involving a single shared project and multiple variable bindings (automation required).
• Con: classpath variable names must be agreed on across components.
• Con: unable to view code for a component that is not a prerequisite (requires loading an additional project).
• Con: unable to browse code for prerequiste components in proper content (code completion).
• Note: uses classpath variables whose value is either a project or a library (in R1.0, JDT core supports these but JDT UI does not expose).

project P1 (shared)
    build classpath = source /P1/C1
    output folder /P1/bin
|
project P2 (shared)
    build classpath = source /P2/C2; P1_LIB
    output folder /P2/bin
|
project P3 (shared)
    build classpath = source /P3/C3; P1_LIB; P2_LIB
    output folder /P3/bin
|
project P4 (shared)
    build classpath = source /P4/C4; P2_LIB
    output folder /P4/bin

Unshared Proxy library projects

• Team member's workspace contains a project for each component.
• Main project is in source and is shared via the repository.
• All other projects in workspace are proxy library projects (same name as source project; contains no source code; not shared via repository).
• Each team members downloads latest binary build to c:\temp\build\ or some such and sets classpath variable BUILD = c:\temp\build\
• Build classpath exports libraries relative to BUILD classpath variable.
• Use of classpath variable allows each developer to choose where to install the build.
• Build classpath references individual dependent projects by name.
• Pro: able to browse code for all components in context.
• Pro: easy to switch to another binary build.
• Pro: switch to patching involves adding source and output folders to non-shared project, populating with selected source files, and building; no other classpaths need to be changed.
• Pro: switch to development involves loading an additional shared project into workspace over top of non-shared proxy library project; no classpaths need to be changed.
• Pro: easy to upgrade to another binary build (unzip build in place, and close and reopen project in workspace to force a refresh).
• Con: initial workspace setup involving multiple non-shared and shared projects (automation required; e.g., preconstructed base workspace).
• Con: project holding component under development is lost in sea of proxy library projects.
• Pro: no restriction on number of jars per project.
• Note: the exported libraries are external to the project.
• Note: uses library projects (proposed for R2.0).

project P1 (not shared)
    library project
    build classpath = export BUILD/c1.jar+BUILD/c1src.zip
&
project P2 (not shared)
    library project
    build classpath = export BUILD/c2.jar+BUILD/c2src.zip; project P1
&
project P3 (not shared)
    library project
    build classpath = export BUILD/c3.jar+BUILD/c3src.zip; project P1; project P2
&
project P4 (not shared)
    library project
    build classpath = export BUILD/c4.jar+BUILD/c3src.zip; project P2

None of these projects are shared; however, they have the same names as the source projects in repository. This means that any of the library projects can be replaced by loading the corresponding source project from the repository. (None of the other projects in the workspace need to change.)

project P1 (shared)
    build classpath = source /P1/C1
    output folder /P1/bin
|
project P2 (shared)
    build classpath = source /P2/C2; project P1
    output folder /P2/bin
|
project P3 (shared)
    build classpath = source /P3/C3; project P1; project P2
    output folder /P3/bin
|
project P4 (shared)
    build classpath = source /P4/C4; project P2
    output folder /P4/bin

Regarding automation, the centralized build could create a simple XML document describing the collection of proxy library projects for the workspace. Given this document, a special purpose plug-in could be written that would create (or modify existing) unshared proxy library projects in the workspace.

<projects>
    <project name="P1">
        <natures
            <nature id="org.eclipse.jdt.core.javanature"/>
        </natures>
        <libraries>
            <classpathentry kind="var" path="BUILD/c1.jar" sourcepath="BUILD/c1src.zip" export="true"/>
        </libraries>
    </project>
    <project name="P2">
        <natures
            <nature id="org.eclipse.jdt.core.javanature"/>
        </natures>
        <libraries>
            <classpathentry kind="var" path="BUILD/c2.jar" sourcepath="BUILD/c2src.zip" export="true"/>
            <classpathentry kind="project" path="/P1"/>
        </libraries>
    </project>
    <project name="P3">
        <natures
            <nature id="org.eclipse.jdt.core.javanature"/>
        </natures>
        <libraries>
            <classpathentry kind="var" path="BUILD/c3.jar" sourcepath="BUILD/c3src.zip" export="true"/>
            <classpathentry kind="project" path="/P1"/>
            <classpathentry kind="project" path="/P2"/>
        </libraries>
    </project>
    <project name="P4">
        <natures
            <nature id="org.eclipse.jdt.core.javanature"/>
        </natures>
        <libraries>
            <classpathentry kind="var" path="BUILD/c4.jar" sourcepath="BUILD/c4src.zip" export="true"/>
            <classpathentry kind="project" path="/P2"/>
        </libraries>
    </project>
</projects>

Shared proxy library projects

In the previous approach to using proxy library projects, these projects were not under VCM. In this variant of it, the proxy library projects are obtained from a repository as well. We will assume that the proxy library projects are stored in a different repository from the source projects. By assuming they're in a separate binary repository, it is easy to use the same project names for both source and binary forms (doing so in the same repository would required introducing non-standard binary-only and source-only branches in the project version histories).

• Team member's workspace contains a project for each component.
• Main project is in source and is shared via the repository.
• All other projects in workspace are proxy library projects; same name as source project, but in a different repository.
• Each proxy library project contains and exports one or more binary jar libraries (plus attached source zips) equivalent to the source.
• Build classpath references individual dependent projects by name.
• Pro: able to browse code for all components.
• Pro: easy to switch to another binary build.
• Pro: switch to patching involves adding source and output folders to shared proxy library project, populating with selected source files, and building; no other classpaths need to be changed.
• Pro: switch to development involves loading shared source project into workspace over top of shared proxy library project with the same name; no classpaths need to be changed.
• Pro: easy to upgrade to another binary build (by catching up with binary stream in repository).
• Pro: no restriction on number of jars per project.
• Pro: Proxy library projects are also under VCM.
• Con: initial workspace setup involving multiple shared projects (automation possible).
• Con: Additional effort of building and maintained proxy library projects in a separate repository (automation possible, perhaps as part of centralized build process).
• Con: project holding component under development is lost in sea of proxy library projects.
• Note: the exported libraries are internal to the project.
• Note: uses library projects (proposed for R2.0).

project P1 (shared via binary repository)
    library project
    build classpath = export /P1/c1.jar+BUILD/c1src.zip
&
project P2 (shared via binary repository)
    library project
    build classpath = export /P2/c2.jar+BUILD/c2src.zip; project P1
&
project P3 (shared via binary repository)
    library project
    build classpath = export /P3/c3.jar+BUILD/c3src.zip; project P1; project P2
&
project P4 (shared via binary repository)
    library project
    build classpath = export /P4/c4.jar+BUILD/c4src.zip; project P2

The proxy library projects have the same names as the source projects, but are stored in a separate repository.

project P1 (shared via source repository)
    build classpath = source /P1/C1
    output folder /P1/bin
|
project P2 (shared via source repository)
    build classpath = source /P2/C2; project P1
    output folder /P2/bin
|
project P3 (shared via source repository)
    build classpath = source /P3/C3; project P1; project P2
    output folder /P3/bin
|
project P4 (shared via source repository)
    build classpath = source /P4/C4; project P2
    output folder /P4/bin

Stub projects

This is a variation on unshared proxy library projects that makes different tradeoffs. We call these stub projects. In particular, the ability to browse a prereqisite component is traded for the ability to have only a minimal set of extra library projects in the workspace. The only difference is that stub projects do not require other projects (whereas proxy library projects did require other projects). This allow a workspace to get by with a less than complete set of stub projects. Creating a stub project never forces you to create other stub projects.

• Main project is in source and is shared via the repository.
• That project's required projects are represented in workspace by stub projects (same name as source project; not shared via repository).
• Stub projects are library projects (no source code).
• A stub project only represents a project's outward appearance to other projects.
• Stub projects never require any other projects.
• Each team members downloads latest binary build to c:\temp\build\ or some such and sets classpath variable BUILD = c:\temp\build\
• Build classpath exports libraries relative to BUILD classpath variable.
• Use of classpath variable allows each developer to choose where to install the build.
• Build classpath references individual dependent projects by name.
• Pro: stub projects are needed only for immediate prerequisites of main project.
• Con: unable to view code for a component that is not a prerequisite (requires loading an additional project).
• Con: unable to browse code for prerequiste components in proper content (code completion) because stub does not provide sufficient context.
• Con: patching only via switching to active development.
• Pro: switch to development involves loading an additional shared project into workspace, possibly over top of stub project; no classpaths need to be changed.
• Pro: easy to upgrade to another binary build (unzip build in place, and close and reopen project in workspace to force a refresh).
• Con: initial workspace setup involving multiple shared projects and stub projects; ongoing need for additional stub projects when new source projects are loaded (automation required).
• Pro: no restriction on number of jars per project.
• Fatal flaw (!): limited able to compile against a stub project due to lack of sufficient context Note: the exported libraries are external to the project.
• Note: uses library projects (proposed for R2.0).

project P1 (shared)
    build classpath = source /P1/C1
    output folder /P1/bin
|
project P2 (shared)
    build classpath = source /P2/C2; project P1
    output folder /P2/bin
|
project P3 (shared)
    build classpath = source /P3/C3; project P1; project P2
    output folder /P3/bin
|
project P4 (shared)
    build classpath = source /P4/C4; project P2
    output folder /P4/bin

The corresponding stub projects (note the absence of required projects):

project P1 (not shared)
    library project
    build classpath = export BUILD/c1.jar+BUILD/c1src.zip

project P2 (not shared)
    library project
    build classpath = export BUILD/c2.jar+BUILD/c2src.zip

project P3 (not shared)
    library project
    build classpath = export BUILD/c3.jar+BUILD/c3src.zip

project P4 (not shared)
    library project
    build classpath = export BUILD/c4.jar+BUILD/c3src.zip

For example, a developer working on P3 would need stub projects for P1 and P2.

project P3 (shared)
    build classpath = source /P3/C3; project P1; project P2
    output folder /P3/bin
&
project P2 (not shared)
    library project
    build classpath = export BUILD/c2.jar+BUILD/c2src.zip
&
project P1 (not shared)
    library project
    build classpath = export BUILD/c1.jar+BUILD/c1src.zip

If they then decided to develop P1 as well, they would replace P1 by the souce project. Their workspace would now look like:

project P3 (shared)
    build classpath = source /P3/C3; project P1; project P2
    output folder /P3/bin
&
project P2 (not shared)
    library project
    build classpath = export BUILD/c2.jar+BUILD/c2src.zip
&
project P1 (shared)
    build classpath = source /P1/C1
    output folder /P1/bin

The fatal flaw is clear in the case of a workspace containing P4 and a stub for P2.

project P4 (shared)
    build classpath = source /P4/C4; project P2
    output folder /P4/bin
&
project P2 (not shared)
    library project
    build classpath = export BUILD/c2.jar+BUILD/c2src.zip

If, for example, a class in C4 subclasses a class in C2 which subclasses a class in C1, then the compiler will need to get its hands on the class in C1. Unfortunately, neither the source code nor binary for C1 is anywhere to be found.

Assorted Other Scenarios

A couple of other scenarios can be constructed using a combination of the techniques employed above:

• Unshared proxy library projects with different project names.

Use a classpath variable per component; bind it initially to the name of the proxy library project; rebind to name of the source project.
• Pro: Allows side-by-side proxy library project and source project.
• Con: Uses both classpath variables and proxy library projects.
• Shared proxy library projects with different project names.

Same as preceding except the proxy library projects can be shared. Use a classpath variable per component; bind it initially to the name of the proxy library project; rebind to name of the source project. Since the proxy library project and source projects have different names, there are no VCM anomalies.
• Pro: Allows side-by-side proxy library project and source project.
• Pro: Proxy library projects are also under VCM.
• Con: Uses both classpath variables and proxy library projects.
• Shared proxy library projects with same project names and same repository.

The proxy library projects can use the same name as the source project and be maintained under VCM in the same repository as the source projects. Note that this is a non-standard arrangement for a VCM project. The source and binary versions of a project need to be thought of as two permanently separate branches in the version history: the source versions contains source code but no binaries, whereas the binary versions contains binaries but no source code. The differences are also reflected in the project's .classpath file. In Eclipse, this can be achieved through the use of a binaries-only stream separate from the usual stream in which the source is maintained. In order to avoid generating unwanted outgoing or incoming changes when switching from binary to source, proceed as follows: unshare from binary stream; delete from workspace; load and share from source stream.
• Pro: Proxy library projects are also under VCM.
• Pro: One repository holds everything.
• Con: Project has non-standard dual version history.
• Con: Separate binaries-only stream means separate branch in version history.

Discussion

The "Source for everything" approach is best for developers who are all involved in the joint active development of all of the components. There is a limit to how large it will scale, since the cost of recompiling everything from source increases method the number and size of components. It is not recommended in situations where many of the components are not under active development; it will be more efficient to use pre-compiled binary libraries for the static components.

Approaches involving classpath variables do not have much to recommend them. They share most of the disadvantages (and none of the advantages) of using proxy library projects.

The "Proxy Library Projects" workspace setup allows each developer to work on their assigned component while providing ready access to all other components. The arrangement is flexible in allowing easy switching from using a component to patching it (or to actively developing it). The two setups outlined in detail show how the proxy binary projects can be constructed from a downloadable binary build or obtained from a version-managed repository. Each has there pluses and minuses. The main drawback of using unshared proxy library projects is the significant task of setting up a workspace in the first place. This task would have to be automated by some means; it would too tedious and error prone to have each developer create a workspace from scratch. For shared proxy library projects, the tedium is in creating new versions of the projects in their repository. This task could be automated as part of the centralized build process.

The "Stub Projects" workspace setup allows each developer to work on their assigned component while providing placeholders for other components that it depends on. The arrangement is flexible in allowing easy switching from using a component to actively developing it by replacing the placeholder. Unfortunately, this approach is fatally flawed because a compiler might not have enough information to compile. The other drawbacks are their limited useful of the stub projects (can't browse the component; can't patch the component), and the ongoing need to create additional proxy projects to fill in for the missing prerequisites as real source projects get added to the workspace.

Document History

15:45 Wednesday September 5, 2001 - first version sent for comments
13:30 Thursday September 6, 2001 - revised for first round of comments
19:40 Wednesday September 12, 2001 - automation for unshared proxy library projects
10:15 Wednesday September 26, 2001 - added stub projects
17:00 Wednesday September 26, 2001 - documented fatal flaw with stub projects