Listeners and builders in a concurrent world

In the context of the work on improving platform responsiveness, there are some proposed changes to how resource change listeners and incremental project builders operate. This document describes and motivates these changes. The existing (Eclipse 2.1) behavior of listeners and auto-build will be described, along with a description of how this will change in Eclipse 3.0. Finally, this document includes a description of how existing implementations might be impacted and how they can adapt to the new behavior.

Overview and motivation

The principal proposed change for release 3.0 is that auto-build will be called in a separate thread from which the workspace change occurred. This means builders will run concurrently with the thread that caused the change, and there will be no guarantee at all about precisely when builders will be called. The only guarantee is that all builders will eventually be run when auto-build is turned on.

There are a number of motivations for this change. First, there is a need to minimize the duration for which the workspace lock is held by an operation. In Eclipse 2.1, every workspace operation had an automatic resource change notification and auto-build cycle attached to the end of it. The operation did not complete until all notifications and auto-builds had finished. For the entire duration of this cycle, the workspace was locked to prevent concurrent changes in other threads. This long term exclusive locking must be reduced if there is any hope to obtain greater concurrency in Eclipse 3.0. One way to shorten the duration of this locking is to split up the operation itself and the auto-build into separate units of work.

In addition, this rigid notification and build strategy resulted in poor performance if clients neglected to batch groups of changes into a single operation (via IWorkspaceRunnable). Decoupling the workspace operation from the build phase allows us to employ strategies to limit the number of builds, and to lower the priority of builds to give precedence to other workspace changing operations.

Another change that will be made to improve responsiveness is to add notifications during long running operations. An upper bound will be introduced on the delay between a workspace change and the notification. This means that even if an operation takes several minutes, it will be reporting incremental resource changes periodically throughout the operation. The increased notifications will act as an added feedback mechanism to show the user that progress is being made on an operation. This also allows the user to begin working with some of the output of the operation before it completes. For example, a user will be able to begin editing imported files before the import is complete, if the import is taking a long time.

Old behavior of listeners and auto-build

Every workspace changing operation or group of changes batched with IWorkspaceRunnable had the following structure prior to Eclipse 3.0:

1. PRE_DELETE or PRE_CLOSE notification if applicable
2. Perform the operation
3. PRE_AUTO_BUILD notification
4. If auto-build is on, perform incremental workspace build
5. POST_AUTO_BUILD notification
6. POST_CHANGE notification

• pre- and post- build notifications occurred regardless of whether auto-build was on.
• pre- and post- build events contained a partial delta for the changes up to that point. Their deltas were always a subset of the delta supplied to POST_CHANGE listeners.
• pre- and post- build listeners are allowed to modify the workspace, but other listeners cannot.

Overview of changes for listeners in Eclipse 3.0

With the proposed changes for Eclipse 3.0, steps 3-5 in the above structure are removed from the operation. The resulting picture looks like this:

1. PRE_DELETE or PRE_CLOSE notification if applicable
2. Perform the operation
3. POST_CHANGE notification

Periodically, the platform will perform a background workspace build operation. The exact timing of when this build occurs will not be specified. The structure of the build operation will look like this:

1. PRE_AUTO_BUILD notification
2. If auto-build is on, perform incremental workspace build
3. POST_AUTO_BUILD notification
4. POST_CHANGE notification

• POST_CHANGE listeners receive notification of absolutely all resource changes that occur during the time they are registered. This includes changes made by builders, and changes made by other listeners.
• PRE_AUTO_BUILD listeners receive notification of all resource changes except changes made by builders and resource change listeners.
• POST_AUTO_BUILD listeners receive notification of all resource changes except changes made by other POST_AUTO_BUILD listeners.

Overview of changes for builders in Eclipse 3.0

The proposed changes for incremental project builders are conservative. The task of an incremental builder is to reconcile an arbitrary set of changes to a densely interconnected web of source files to produce an output that is a consistent representation of the source. This is a difficult enough task to achieve in a single-threaded environment that to attempt it in a concurrent environment would be courting disaster. On the other hand, building can take a long time, and there is a very high value in moving this work to a non-blocking background thread to allow the user to continue working.

With these factors in mind, the proposal is to move auto-build into a background thread, but to preempt the build when another operation attempts to modify the workspace. A typical work flow would go like this:

1. A user modifies a file and saves the change.
2. Auto-build starts.
3. The user modifies the file again and attempts to save.
4. The auto-build started in step 2 is halted at the nearest convenient stopping point.
5. The file modification is allowed to proceed.
6. Auto-build starts running again.

This new strategy is designed to have no impact on existing builder implementations. In essence, this approach is equivalent to a mode where auto-build is off, but a background thread is performing an incremental build periodically. In other words, since this strategy (apart from the preemption) can be implemented using existing API, it should have no impact on existing builders. No new API is proposed for builders, and no API changes are planned.

Summary of proposed API changes

From a binary compatibility view, this proposal includes no new API, and no API changes. The changes are 100% binary compatible with all prior Eclipse releases. There are two proposed changes to the API contract of resource change listeners:

1. It will no longer be guaranteed that all resource changes that occur during the dynamic scope of an IWorkspaceRunnable will be batched in a single notification. This mechanism can still be used for batching changes to avoid unneccessary builds and notifications, but the platform may decide to perform a notification during the operation if sufficient time has elapsed. Note that it was previously possible for resource change notifications to occur during the invocation of an IWorkspaceRunnable via the IWorkspace.checkpoint API method. Therefore, this API change is not likely to be a breaking change for existing clients. It is equivalent to the platform deciding to call IWorkspace.checkpoint periodically during a long running operations.
2. The javadoc of IResourceChangeEvent states that POST_CHANGE events happen immediately after POST_AUTO_BUILD events. This will no longer always be true. The converse will still be true: POST_AUTO_BUILD events will always be followed immediately by a POST_CHANGE event. To boil it down: auto-build events still occur at the same time in relation to builds, they still receive exactly the same information, but they just occur less frequently.

How clients might be impacted

From the viewpoint of a listener of POST_CHANGE notifications, this new structure has no effect at all. This new structure is equivalent to a mode where auto-build is turned off, and the user is performing a manual workspace build once in awhile. Most importantly, POST_CHANGE listeners are still guaranteed to receive a delta that contains all changes made since the last notification.

For listeners to auto-build events, the new structure has some signficant differences. Primarily, it will no longer be true that upon completion of a workspace changing operation, that auto-build listeners will have been notified. Client code that makes assumptions about when auto-build listeners run are likely to be broken by this change. For example, if an auto-build listener is updating a domain model to reflect changes to the workspace, then this update might not have happened when the workspace changing operation returns.

It is worth noting that only UI-level code can be affected in this way. Core-level code that is called via API may be called within the scope of an IWorkspaceRunnable, so it can never be sure about when resource change listeners will be called. The suggested fix for this breakage is to use POST_CHANGE instead of build listeners if it is necessary to have notification occur before the operation completes.

Another potential breakage is in clients that make assumptions about the ordering of resource change listeners. Previously, PRE_AUTO_BUILD, POST_AUTO_BUILD, and POST_CHANGE events occurred in a strict linear order. If listeners in POST_CHANGE assume that the auto-build events have already fired, they may be broken.

Status and schedule

All described changes have been implemented. All work is currently in branch "Bug_36957" of project org.eclipse.core.resources. Basic sanity checking of the SDK reveals no significant problems, and some core team members are self-hosting with it. All platform core test suites now pass with background auto-building. There are some failures in JDT core test suites where they make assumptions about the timing of auto-build deltas. The JDT core team is investigating this. The hope is to release these changes to integration builds before the 3.0 M3 milestone.

Go back to concurrency home.