Workspace Structure

Original work item: "Make the build path less sensitive with regard to whether a project is a source or a binary project. This is also related to patching (i.e., adding a source folder to a binary project)."

Related issue: support for extension directories containing many jars.

There was much discussion. The underlying issue is how to structure workspaces for significant Java development efforts. The Eclipse Project is an example of such an effort, so the question is not merely of passing interest. Since much of what is being developed in the Eclipse Project are plug-ins, the question also touches on PDE's domain. However, some aspects of the problem affect all significant Java development efforts and not just ones for developing plug-ins. So we begin to investigating those.

Problem: Close Collaboration between Adjacent Teams

The following is an attempt to capture a familiar problem in a general way that does not presume exactly how developers' workspaces are to be structured. Consider the case of two components, with PUI dependent on Core (imagine that Core is the core component, and UI is the corresponding UI component built atop it). Now imagine that Core and UI are under active development by two teams, with ownership split along component lines. For maximum convenience to both teams, the teams would like to be able to do the following:

• Each team would like to be able to work in a workspace that contains the source code for their component and the binary equivalent (with attached source) for all other components.
• Occasionally (e.g., during debugging) each team would like an easy way to workaround (or perhaps induce) problems in another component.
• Occasionally each team would like an easy way for members to "join the other team" and more actively work on another component.

Problem: Large Numbers of Libraries

Referencing a project's libraries in another project: The project I'm working on uses a fairly large number of libraries (about 20 jar files, IIRC). I had hoped I'd be able to create one project which used these libraries, and make all other projects have that project on their build-paths to bring in the libraries - that way I wouldn't need to change *every* project every time I added or removed a library. (It's not a common operation, but even so...) Unfortunately, I can't get this to work - putting project A on the classpath of project B only seems to put project A's individual source/class files on the classpath for project B. I've tried adding as an external jar and importing the jar file into the project, and had no joy with either :(

project P1
    build classpath = source /P1/src; library lib1.jar; library lib2.jar; ...; library lib20.jar
    output /P1/bin
project P2
    build classpath = source /P2/src; library lib1.jar; library lib2.jar; ...; library lib20.jar
    output /P2/bin

Review: Required Projects

• The class files in the binary output directory of R are included, like a library, on the effective build classpath of P. P's build classpath indicates the order of this library relative to others. Other library (and project) entries on the build classpath of R have no bearing on P; they are only consulted to browse or build R.
• The names of required projects are recorded in the .classpath file, which is under VCM. They are therefore shareable with other developers.
• Projects are presented as top level elements in the standard packages view. A required project is not presented in the packages view as a child of the project(s) that requires it.
• A project's list of required projects (a JDT core notion) is used to computed the default value for the project's list of project references (a workspace notion).
• Missing required projects and libraries, and unbound classpath variables, generate problems reported against the project resource itself. These kind of problems are detected by the Java model (rather than the Java builder).

Review: Classpath Entries

• Source folder entry: denotes a project folder containing .java files. The source folder must be contained inside the project where this classpath entry is used. Source folder entries are generally used to componentize the content of a given project (e.g. org.eclipse.jdt.core/Eclipse Java Compiler/). The action of building a Java project will lead to populating the project output folder with .class files corresponding to the .java files which are contained in all the source folders present on this project classpath. Note that the project root can itself be used as a source folder (in which case the output folder is also the project itself).
• Library entry: denotes a binary JAR archive or binary folder. In case of pointing at a JAR, the entry can also define a source attachment recommendation (path to source archive, and path of source root inside this archive, e.g. "c:/jre/src.jar" + "/src").
• Project entry: denotes a required project. When building, the required project is built before the dependent project, and contributes its entire output folder (i.e. all its produced .class files). Thus a project never directly contributes its source files when building. However, for all source-based functionalities (code assist, search, ...), a required project directly contributes its sources (i.e. all of its source folders are exposed) so that a build action is not mandatory to obtain accurate information; in particular, when doing intensive code reorganization, code assist and search will still perform accurately even if auto-build is turned off.
• Variable entry: indirect reference to either a library or a required project. A variable entry uses a variable path of the form %variableName%[%pathSuffix%]. The variable name will be substituted with its actual value, which is a workspace defined constant. The path suffix is an optional suffix which can be appended to the variable value. PDE uses a global variable to represent the Eclipse home directory, and path suffixes to reach the well-known libraries (e.g.    "ECLIPSE_HOME/org.eclipse.jdt.core/jdtcore.jar"). Note that in a consistent manner with library entries, variable entries can be provided with a variable source attachment recommendation.

Investigation: Make Classpath Variables More Powerful

One idea would be to allow variables to be bound to a list of paths instead of a single path. This would allow, for example, a single classpath variable "BASE_LIBS" to be bound to the list of the paths of the library jars lib1.jar; lib2.jar; ...; lib20.jar. Each project referencing this classpath variable would thereby gain access to all the libraries:

project P1
    build classpath = source /P1/src; BASE_LIBS
    output /P1/bin
project P2
    build classpath = source /P2/src; BASE_LIBS
    output /P2/bin

Since classpath variable bindings are local to the workspace, there is no obvious automatic way by which these classpath variables would get their bindings. The bindings would have to be configured for each workspace; loading bindings from a file, or initializing them via a script, are feasible options..

The classpath variables currently bind to paths, and classpath entries can contains paths that begin in a variable. Changing variables to bind to a list of paths would be a major change, and would likely require reworking much of existing API. We generally agreed that we would not pursue this approach since it seems somewhat unlikely to solve much of the problem.

Proposal: Explicitly Export Libraries from Required Projects

The proposal is to extend the required projects mechanism to allow a required project to contribute more than just its binary output folder. Rather, a project would be able to indicate that any of its libraries are to be exported. Exported libraries also become available to other projects in the workspace that list this project as a required project.

In the above example, the customer could instead have an arrangement like:

project P1
    build classpath = source /P1/source; project PLib
    output /P1/bin
project P2
    build classpath = source /P2/source; project PLib
    output /P2/bin
project PLib
    build classpath = source /PLib/source; export library lib1.jar; ...; export library lib20.jar
    output /PLib/bin

The modified semantics of a project P with a required project R are as follows:

• There is an export flag associated with each library explicitly included on the build classpath of a project. This flag is meaningful for both library classpath entries (both internal and external), and required projects (it is not associated with source and output entries.) The export flags are recorded in the .classpath file, which is under VCM. These flags are therefore shareable with other developers. (These flags can be added in a way that does not invalidate existing R1.0 .classpath files.)
• The class files in the binary output directory of R, along with any libraries explicitly exported on R's build classpath, are included as libraries on the build classpath of P. P's build classpath indicates the placement of this library relative to others; the output folder is always first, with the exported libraries and projects included in the user-specified order. Exported required projects are expanded inline. This ordering supports scenarios where the output folder contains patches to the libraries. Non-exported library entries and project entries on the build classpath of R have no bearing on P.
• The names of required projects are recorded in the .classpath file, which is under VCM. They are therefore shareable with other developers. (Unchanged.)
• Projects are presented as top level elements in the standard packages view. A required project is not presented in the packages view as a child of the project(s) that requires it; nor are the required project's exported libraries.
• A project's list of required projects (a JDT core notion) is used to compute the default value for the project's list of project references (a workspace notion). (Unchanged.)
• Missing required projects and libraries, and unbound classpath variables, generate problems reported against the project resource itself. These kind of problems are detected by the Java model (rather than the Java builder). (Unchanged.)
• Classpath problems pertaining to libraries exported from a required project are reported only once, against the project closest to the problem. The problematic entries are elided from the effective classpath of dependent projects.

project P
    build classpath = source /P/source; project R; library plib.jar
    output /P/bin
project R
    build classpath = source /R/source; export library rlib1.jar; library rlib2.jar; export project Q; export library rlib3.jar
    output /R/bin
project Q
    build classpath = source /Q/source
    output /Q/bin

    build classpath = source /P/source; library /R/bin; library rlib1.jar; library /Q/bin; library rlib3.jar; library plib.jar

Exporting External Libraries

Should projects be allowed to export arbitrary libraries, or should exported libraries always be internal to the project?

Restricting exports to internal libraries ensures that a project that is to be used by other projects is somewhat self contained. If you load such a project from a repository, you are guaranteed all the libraries it exports will be contained therein.

On the other hand, unrestricted exports are more flexible, and allow a project to export a collection of libraries that are not necessarily contained within the project's resource tree. (However, it is unclear whether this additional flexibility would be useful.)

We opted to allow unrestricted imports, but decided to simplify the error processing by only reporting problems against the project with the missing library (or required project) explicitly on its classpath. Any missing entries would simply be omitted from the effective classpath calculation.for dependent projects.

Classpath Variables

A project's build classpath may include references to classpath variables that get bound to libraries (or other projects). How do exports and variables interact?

The export flag could be associated either with the variable reference or with the variable binding. If the export flag is associated with the variable reference, it would indicate exporting whatever library the classpath variable happened to be bound to on a given occasion. If the export flag is associated with the variable binding itself, the library the classpath variable happened to be bound to on a given occasion would be exported conditionally on the flag in the binding. The proposal is to go with the former (export flag with variable reference) since its semantics are somewhat simpler and allow the export flag to be shared with other team members (the bindings of classpath variables are not shareable).

Exporting Auxillary Libraries

Note that explicit exports also address another problem that arises when a project needs to export a pre-built library in addition to the results of compiling its source files (the Eclipse debugger's jdi.jar is a fine instance of this). In Eclipse 1.0, only the class files in the project's binary output folder are exported to dependent projects. This proposal allows a project to export any number of additional libraries as required.

Exporting Projects

By exporting other projects, a project can consolidate and concentrate the outputs from several other projects:..

project P1
    build classpath = source /P/source; export library /P1/lib1.jar
    output /P1/bin
project P2
   build classpath = source /P2/source; export library /P2/lib2.jar
   output /P2/bin
project P1andP2
   build classpath = source /P1andP2/source; export project P1; export project P2
   output /P1andP2/bin

build classpath = library /P1andP2/source; library /P1/bin; library /P1/lib1.jar; library /P2/bin; library /P2/lib2.jar
output /P1andP2/bin

Proposal: Admit Library Projects which Contain No Source Code

One objection to this whole approach is that the Java project is being hijacked. In the original design, a Java project is a buildable container of Java source code. The addition of exported libraries starts to turn the project into something more general. If we are comfortable with this general trend, there are ways that the design can embrace it more whole-heartedly.

The above proposal to expand the semantics of required project creates a new role for a Java project as a container of libraries. This notion of library project is rounded out by allowing Java projects without source or output folders. The build classpath of a library project orders the list of exported libraries and projects (and provides the additional context required for browsing the project itself).

In our example, the library project could be expressed more directly without having to postulate source and output folders which are completely unmotivated in this case:

project PLib
    build classpath = export library lib1.jar; ...; export library lib20.jar

A library project is recognizable simply by the absence of source folders on the build classpath. (We considered adding a new flag to the project to explicitly mark library projects, but decided this was not necessary and would be confusing.)

Note that library projects are not "buildable" in any meaningful sense since they lack source code to compile (and without source code there is no pressing need for a binary output folder). However, there still needs to be a mechanism for detecting and reporting errors in the build classpath for library projects. Except for such checks, the Java incremental project builder should ignore library projects.

The following kinds of entries on the build classpath of a library project would serve these purposes:

• Source folders - a library project has no source folders, by definition.
• Output folders - not needed for a library project.
• Unexported libraries - provides context when browsing this project.
• Unexported required projects - provides context when browsing this project.

The workspace must compute an input delta even for incremental project builders that ignore the delta they are handed. We considered whether we should go one step further and remove the Java builder from a library project's list of incremental project builders. This would make it crystal clear that there is no Java building going on, and it would reduce workspace memory footprint because the workspace would not need to remember the shape of the resource tree at the time of the last build. However, if the builder was removed for library projects, there would need to be mechanism to add it if the classpath was changed to include a source project. We decided that we should simply avoid the additional hassles and leave the Java builder installed for all Java projects.

Background

From Philippe Mulet 08/31/2001 12:13 PM

I also think this has to be a basic mechanism at the JavaCore level, which the PDE can then surface to plugin writers.
I would like to see this one addressed asap, so that it is possible to share projects transparently at least amongst us.

=======

From Erich Gamma on 08/31/2001 12:43 PM

Here is some more input on our favorite topic "make the build class path less sensitive
with regard to whether a project is a source or binary project".

Our conclusion in SNZ was that this boils down to a PDE project layout issue (binary
projects etc). However, the EC discussion from below illustrates a use case for allowing to export
JARs in addition to the output folder from a project, that is independent of PDE.
The scenario is similar to the WSAD scenario with extension dirs. If we support to
contribute contained JARs from a project then this problem would be addressed.

We should therefore reconsider the solution independent of the PDE issue.
If I do that then I come to the conclusion that the proposed mechanism is valuable
and I suggest to go ahead with specifying it in detail.

Thoughts?

=======
From Jon Skeet <skeet@pobox.com> on Eclipse Corner 20.8.2001

Referencing a project's libraries in another project:
The project I'm working on uses a fairly large number of libraries
(about 20 jar files, IIRC). I had hoped I'd be able to create one
project which used these libraries, and make all other projects have
that project on their build-paths to bring in the libraries - that way I
wouldn't need to change *every* project every time I added or removed a
library. (It's not a common operation, but even so...) Unfortunately, I
can't get this to work - putting project A on the classpath of project B
only seems to put project A's individual source/class files on the
classpath for project B. I've tried adding as an external jar and
importing the jar file into the project, and had no joy with either :(

Document History