Eclipse IDE Meta-tooling Platform ("The Eclipse IMP")

The project has been created. Please visit the project page.

Introduction

The Eclipse IDE Meta-tooling Platform is a proposed project under the Eclipse Technology Project.

This proposal is in the Project Proposal Phase (as defined in the Eclipse Development Process document) and is written to declare its intent and scope. This proposal is written to solicit additional participation and input from the Eclipse community. You are invited to comment on and/or join the project. Please send all feedback to the http://www.eclipse.org/newsportal/thread.php?group=eclipse.technology.imp newsgroup.

Background

Eclipse provides a very high-functionality IDE for Java, as well as for several other languages (e.g. C/C++, Python, Ruby). At this point, however, the cost of developing an IDE for a new language remains prohibitively high. This is due largely to the fact that although Eclipse offers a rich suite of building blocks for constructing a wide variety of applications, the set of APIs intended specifically for building IDEs is largely tied to specific languages. This precludes code reuse even where common concepts and language processing idioms already exist.

Moreover, building an IDE for Eclipse is an extremely knowledge-intensive task, requiring intimate understanding of hundreds or even thousands of interfaces and implementation classes. As a result, the learning curve is daunting, and there is not nearly enough guidance available to direct the IDE developer to the relevant interfaces and reusable implementation classes.

Description

The Eclipse IMP's purpose is to develop a set of extensible frameworks and exemplary tools built on the Eclipse platform that ease the creation of IDEs for programming languages (new or otherwise) that do not yet enjoy full-featured Eclipse support. Our ultimate goal is to assist language implementers in creating IDEs with a richness and usability on par with the Eclipse Java Development Toolkit (JDT). In the near term, however, we intend to provide support for a significant portion (but not all) of the JDT's functionality. It is our belief that this is a natural next step in the evolution of the Eclipse - doing for programming IDEs what RCP does for rich UI applications.

By accelerating the process of developing new IDEs, we also hope to expand the use of Eclipse to new programming domains, further enhancing the appeal of the Eclipse platform. In fact, we hope to encourage language design experimentation by making it viable for language implementers and their communities to produce full-featured IDEs with more reasonable effort than is presently possible.

Proposed components

  • Service Generators. A set of code-generating wizards and templates that generate stub/initial implementations for various language-specific IDE services, along with any necessary meta-data (e.g. extensions of IMP-defined extension points).
  • IDE Runtime. A class library that encapsulates behavior and appearance common to many IDEs and languages, with hooks (via extension points or abstract methods) for providing language-specific customizations. Example sub-components include views such as the source editor, the outline view, or the program explorer, a dependency tracking mechanism for incremental building, and so on.
  • Language Service Extension Points. A set of extension points that encapsulate the various language-specific IDE service implementations (e.g. parsers, syntax highlighters, outliners, formatters, etc.).
  • Declarative Languages. A set of domain-specific languages that will allow IDE developers to specify such language-specific concepts as language syntax (e.g., for syntax highlighting or outline contents), language semantics (used in providing various other language services), appearance in the IDE of program elements, program transformations (e.g. refactorings), various program analyses, and so on. This will include full-featured IDE support for the declarative languages themselves.
  • Parser Generator IDEs. Since much of IDE development activity revolves around the language grammar, this component provides IDE support for various existing grammar specification languages and parser generators, such as LPG, Antlr, or JavaCC.
  • Compiler/Parser Adapters. IMP will not require any particular scanning, parsing, or AST technology. In particular, IMP must be capable of reusing existing compiler front-ends. This has the dual benefit that IDE developers can avoid gratuitously rewriting existing code, and that the IDE and compiler are in complete agreement as to the structure and meaning of every construct. To that end, some adapters will be required to permit IMP components to work with existing parser/compiler generators, such as LPG, Antlr, or JavaCC.
  • AST-based Tools. A key part of refactoring is transforming existing code. Currently, in the JDT this is done by writing verbose Java code to manipulate ASTs and build new AST nodes. A language-independent AST pattern/transformation language would help speed the development of new transformations and make it easier to reason about their correctness. Such a pattern language would also be useful for specifying how to populate search indexes, specifying candidate "quick fixes", as well as for providing the basis for a user-level AST-based search mechanism. We intend to develop a suite of domain-specific languages for accomplishing these and other tasks, all making use of a common AST pattern language.
  • Static Analysis. A set of reusable analysis components (type inference, pointer analysis, call-graph construction, escape analysis, etc.) to be used in constructing refactorings and other tools that rely on deep semantic knowledge of the program. While static analysis is by definition language-specific, the analysis algorithms themselves can share much code, including data structures, graph algorithms, generic solvers, etc. Probably to be built on top of an existing static analysis framework such as WALA.
  • Debug Framework. This component would provide, e.g., a specification language to define the debug-time presentation of the runtime platform (e.g. backtraces, variables, breakpoints, etc), along with a framework to map this specification into implementations of the necessary Eclipse Debug APIs.
  • Process Documentation. Cheat sheets and other forms of documentation to guide the IDE developer through the process.
  • IDE-building Perspective. A perspective helps focus the user's attention on the relevant IDE-building views, actions and commands by making them appear in the toolbar, top-level menus, and so on.

Relationship with Other Eclipse Projects

The Eclipse IMP will be built on top of the Eclipse Platform. Since the JDT represents the desired level of functionality, we intend to work with the JDT team to identify existing components in the JDT that can be generalized and made part of the IMP.

The LTK (Language ToolKit) project provides runtime language support components, which is used for example by the IMP to provide refactoring support. We will investigate which IMP components could be moved to the LTK, which can be used even by IDE developers who do not build their IDE using the IMP meta-tooling.

The Web Tools Platform's Structure Source Editor may also provide components useful for the source editing portion of the IMP runtime.

The DLTK is a related effort, but requires the use of a specific AST class hierarchy to drive much of its functionality, which is not suitable for arbitrary languages, nor for situations in which a compiler front-end already exists. Nevertheless, we intend to investigate what portions of the DLTK's infrastructure can be reused or adapted.

Ed Merks of the EMF project believes that EMF may benefit from enhancing IMP to permit the design and implementation of concrete textual language syntax from existing EMF models. This is based on the observation that an EMF model can be viewed as abstract syntax for a language. The result is that rather than using generic list- and tree-based EMF editor views for both the model and model instances, one can instead use IMP-based language-specific (i.e. model-specific) textual source editors, with all the affordances that IMP provides for textual languages (e.g. syntax highlighting, content assistance, navigation, hover help, etc.).

Organization

We propose this project should be undertaken as a Technology project rather than as part of the Eclipse Platform. The technology is not yet well defined enough in either specification or implementation to be ready for general consumption. As a Technology project, we will have the opportunity for eager adopters to use, critique, and improve the software in a non-disruptive manner before its use in mainline Eclipse Platform development work.

Finally, since Technology sub-projects are not meant to be ongoing, we foresee three possible evolutionary paths for the subproject:

  1. The project is merged back into the mainline Eclipse Platform top-level project.
  2. The project is moved to a permanent state as a Tools Platform project.
  3. The technology proves to be insufficient for Eclipse and the work is set aside.

In all likelihood, some combination of outcomes (1) and (2) will make the most sense for sub-portions of the components developed within this project.

Mentors

  • Rich Gronback, Borland
  • Ed Merks, IBM

Proposed initial committers

Code Contributions

IBM Research is offering to contribute its Eclipse IMP software (previously released as open-source here) as the initial codebase.

Interested parties

  • Borland, Inc. - Artem.Tikhomirov@borland.com
  • Eclipse JDT team - Martin Aeschlimann, Philippe Mulet
  • Oxford University - Oege.de.Moor@comlab.ox.ac.uk
  • Centrum voor Wiskunde in Informatica (CWI) - Paul.Klint@cwi.nl, Jurgen.Vinju@cwi.nl
  • Cornell University - Andrew Myers
  • EPFL, Switzerland - Martin Odersky
  • Delft University - Eelco Visser
  • Eclipse CDT team
  • IBM Rational - Chris Laffra

Tentative Plan

  • Initial Eclipse.org presence in Oct 2007
    • website
    • newsgroup
    • CVS repository, seeded with source code from current (non-public), if IBM source code contributions are accepted
    • Bugzilla repository, seeded with bug reports from current (non-public) Bugzilla repository, if IBM's code contributions are accepted.
  • Initial Eclipse.org release, v1.0: Dec 2007, including:
    • IMP Runtime library including language-independent editor
    • Initial version of meta-tooling wizards, with support for scanning, parsing, syntax highlighting, outlining, source folding, hover help, hyperlinking, content assist, incremental building.
    • Meta-tooling support and IDE support for LPG-based grammar and scanner specifications, with APIs that permit integration with other parser/scanner generators
    • Exemplar IMP-based IDEs for X10, the LPG grammar specification language, Java and JavaScript
    • Cheat sheets guiding/describing IDE development process
    • Initial installation and use documentation, taken from existing (non-public) web pages
  • Second Eclipse.org release, v1.1: June 2008, possibly including:
    • Language embedding
    • Integration with the JAstAdd compiler framework
    • Integration with other parser generators such as ANTLR and/or JavaCup
    • Improved support in meta-tooling and code templates for language inheritance