Building & Running a Language Server with Eclipse Xtext & Theia

In this article, you will learn how to implement a Language Server for an arbitrary DSL using Eclipse Xtext and then run it in an IDE. For the IDE part, we will use Theia, a new open-source initiative to build a framework that allows single sourcing IDEs that run on the desktop as well as in a browser/cloud context. More information on Theia can be found here.

The example project explained in this article can be found on GitHub. It consists of three parts:

1. An Xtext project (the actual language server)
2. A Theia extension
3. A Theia application

Feel free to clone the repository and tinker around, while or after reading this article. The sets of commands listed in the README should bring up a browser IDE similar to the following screenshot:

The Language Server

In case you don’t know, the language server protocol defines a set of messages that an editor can exchange with a language server. A language server is a process that knows a thing about a particular language. Using the LSP, modern compilers can expose advanced language features, such as code completion and find references, in a UI-independent and therefore reusable way. Finally, Xtext is a mature framework to develop compilers and language services and was one of the first tools to support the LSP.

To get started with an Xtext Language server, I created an Xtext project using the Eclipse IDE. In this example, I was only interested in the language server part, so the wizard should not create any editor-specific projects. Also, the build should use Gradle. Therefore, I used the following wizard configuration:

The example DSL allows defining JSON-RPC protocols (sorry, I couldn’t resist ;)). One can declare protocols with requests and notifications, as well as types, which are used as parameters and responses. Here is a simple example file for our DSL:

protocol MyProtocol {
    request someRequest(arg: RequestParam): RequestResponse
    notification hello(msg: string)
}

type RequestParam {
    something: string
}

type RequestResponse {
   answer: string
   time: number
}

Implementing such a language in Xtext is easy since you only need to change the grammar file that was generated by the wizard. Based on that, the language server will be able to provide features like diagnostics, navigation and so on. Xtext also provides hooks to implement code generators, formatters and additional validations. If the defaults don’t fit your needs, you can change any aspect of your language infrastructure on a very fine grained level. The grammar file, describing our DSL can be found here.

Packaging The Language Server

As mentioned before, a language server just is a process that an editor can spawn and then communicate with by sending messages back and forth. To start our language server we need to provide an executable. Xtext is running on the JVM, so we need to make it easy to start a Java process. I used the shadowJar gradle plugin to create a so called ‘uberJar’ for our application that contains all upstream dependencies. So we can run it by executing ‘java -jar uber.jar’.

To apply the shadowJar plugin, we change the 'build.gradle' file in the parent project like this:

buildscript {
  …
  dependencies {
     classpath ‘com.github.jengelman.gradle.plugins:shadow:1.2.4'
     …

…
subprojects {
  …
  apply plugin: 'com.github.johnrengelman.shadow'
…

In addition we need to configure the shadowJar task in the *.ide project’s build.gradle:

shadowJar {
    baseName = 'dsl-language-server'
    classifier = null
    version = null
    manifest {
        attributes 'Main-Class': 'org.eclipse.xtext.ide.server.ServerLauncher'
    }
}

On the root level, you can now execute ./gradlew shadowJar, which should do the build and produce a jar file that contains all upstream dependencies.

The specified main-class is shipped with Xtext and provides a standard way to start a language server that will run all languages that are on the classpath. In other words,with Xtext you can have one language server that can handle multiple Xtext languages. This is important if those languages link against each other. Also, editors usually communicate with a language server process through ‘standard in’ and ‘standard out’ which is the case here, too.

Building a Theia Extension

Theia is the editor we are using in this example. It is implemented in TypeScript and is composed of extensions. The unique thing about Theia is that it runs both as a native desktop app like the traditional Eclipse IDE and in the browser like Eclipse Che or Eclipse Orion. So you need to implement your tool only once and get support for both.

To add support for a particular language within Theia, we build an extension and include it in a Theia application. Theia runs in two parts, a frontend process and one or more backend processes. To register a language server it would usually be enough to build a backend extension, but since the LSP lacks some important configuration (e.g. lexical syntax coloring) we also need to write a frontend extension to provide this.

Dependency Injection

An extension in Theia is an npm package that provides one or more modules for the dependency injection (DI) container (Theia uses inversify.js. Using DI allows extension providers to integrate with other extensions, either by contributing hooks that other extensions can provide implementations for or by providing implementations to other extension’s hooks. Besides, DI makes it easier to test code in isolation.

Backend Extension

The backend extension provides a LanguageContribution, a hook provided by the general languages extension. Have a look at the actual code. As you can see we don’t do much other things than telling Theia that there is a language called “DSL” that can handle *.dsl extensions. Also, we get passed in a connection object that we use to create a server process. That’s all you need to do to setup any language servers in Theia.

Frontend Extension

The frontend extension directly provides configuration to Monaco’s language registry. Monaco is the editor widget that is currently used in Theia and is the editor that also sports Microsoft’s Visual Studio Code. That said the Monaco integration is itself an extension and hides behind an editor abstraction. So other code editors, like e.g. the one from Orion, can be used as well.

The client side configuration provided is not a part of the LSP, which is why it is configured here and not in the language server. The provided information includes tokenizing rules for syntax coloring, bracket information for matching and auto closing, as well as file extensions and mimetypes. Moreover, custom extensions to the language server protocol can be registered in a frontend extension. The Java language server for instance, has some additional features to open code from jars.

Running The App(s)

Finally, we should have a brief look at the actual Theia application. First, to consume our theia-dsl-extension we need to add a dependency in the package.json file. Since we don’t want to publish the extension to npm.org all the time, we use a file dependency:

// package.json
dependencies {
    …
    "theia-dsl-extension": “file:../theia-dsl-extension”
}

Other than that, I simply copied the example app from Theia and added our dsl extensions to the ‘app.ts’ (backend) respectively ‘main.ts’ (frontend). In addition to the composition of the frontend and backend applications, it also come with the needed webpack configuration.

You can now run our new IDE as a browser application with node-based backend, or as a native desktop app using electron using the commands from the README file.

Outlook

This article explained just one way to leverage the LSP with Xtext. The same language server can be integrated in other IDEs and maybe even more interesting in any custom web application. Theia, on the other hand, can run any language server - not only those implemented in Xtext. Even though it is a very young effort, it can already do advanced language support for many different languages, thanks to the LSP. Theia makes heavy use of such protocols in other areas too. The FileSystem API, for instance, is also based on a JSON-RPC protocol, which allows to exchange the frontend and backend implementations at a later point if needed.

The best part is, both Xtext and Theia are open and diverse open-source projects. Any form of contribution is highly encouraged and welcome.