Building Xtext Languages with Maven and Gradle

Xtext languages should be built, tested and deployed continuously, just like any other piece of software. Users expect generators to run inside popular build tools and not just inside the IDE. In this post you will learn how to create a Maven build for your Xtext language and how to consume the resulting Maven artifacts from other projects, both Maven and Gradle.

The example language that we will be building is the BuildDSL from the Seven Languages project. The language itself (i.e. the grammar and tooling) are not important for this exercise. However, it has several interesting properties that allow me to show you most of the challenges you will encounter when building a language:

• It uses Xbase, linking and generating Java code
• It has a standard library that needs to be shipped with the language
• It has UI tests which need some additional setup

The code for the examples is available on Github.

Choose your path

There are two ways of "mavenizing" your language build. The first is the minimalistic approach, which relies on checking in generated code. This means that this Maven build is not able to generate the language infrastructure or clean the output folders by itself. It is just there to create Maven artifacts for others to consume. This approach is very non-intrusive for developers who are used to an Eclipse build. There are two main disadvantages to this. Checking in generated sources will make your diffs pretty hard to read. And if one of your developers has a slightly different setup than everyone else (line endings, encoding, etc.), the build might run on his machine, but fail when he checks in.

The alternative is a Maven build that can generate everything from just the source code. This means you can stop checking in generated code. And since all important configuration is part of the build script, you will get the same result on all platforms. But it also means that you will have to run the Maven build once after checkout in order for your projects to compile. In the end this is a matter of preference and you and your team should decide which suits you better.

The starting point

• Let's fist have a look at the unaltered language projects. You will find them in the language/0-initial folder. I will give you just a quick rundown what each of them does. If you have worked with Xtext before, you should feel right at home.
• builddsl contains the core language infrastructure and code generator
• builddsl.lib is the standard library that is shipped with the language
• builddsl.ui contains the Eclipse UI parts (like content assist)
• builddsl.tests contains all tests for the language, including UI tests
• builddsl.target is a target platform definition for the language. If you don't use one, then your language will be built against whatever is installed in your host workbench. This makes builds highly platform dependent. So every project should have a target platform.
• builddls.repository assembles the plugins into a p2 repository, from which the plugins can be installed

The minimalist approach

Next up we will create the minimal required POMs to build the language with Maven. You can find these projects under language/1-simple. The most important addition is the builddsl.parent project. This is the main entrypoint for Maven. It lists all other projects which should be built and contains common configuration for them. The crucial part is enabling the Tycho Maven plugin.

Tycho is a Maven extension that can read dependency information from Manifest files and fetch these dependencies from p2 repositories. It also teaches Maven which build steps are required for Eclipse plugins. The second important bit of configuration is the target platform definition. We just reference our target project and list the environments for which we want to build. Without specifying these, Tycho would just use the currently running system as the default, which would make the build platform dependent.

The POMs for the individual projects are all very straightforward. Each inherits from the parent and sepcifies a packaging type. Tycho adds packaging types for Eclipse plugins, target platforms and repositories. One of the more interesting POMs is that of the core language plugin. It needs a dependency to the library plugin, even though that dependency is already present in the Manifest.

This is the downside of using Tycho. If you want Maven clients to understand your dependencies, they always need to be in the POM But Tycho does not "enhance" the POM with the dependency information from the Manifest. Luckily, this is the only bit of duplication we need. All Xtext runtime dependencies are already on Maven Central and part of the Xtext plugins for Maven and Gradle, which we will use later.

The other more elaborate POM is that of the tests plugin. You will see that we enable the UI harness so that we can run the content assist tests contained in the project. Sadly, we need to add some platform specific handling for OSX, as otherwise SWT will not run. This is the price you pay if you want your build to run well on all platforms.

Try running mvn install from the parent folder to install the language into your local Maven repository. Deploying the artifacts to some other repository like Maven Central is out of scope for this post, but well documented and no different from any other Maven project. Also, you will find a p2 repository in the /target folder of the builddsl.repository project. You could transfer this to some public server so that other people can add your language to their Eclipse installation.

Using your language

You are now ready to use your language from the client projects. There are two example clients. Each of them contains the same code: A builddsl file which references an Xtend file. This is just to show that referencing existing Java code works as desired. As a bonus, both projects also showcase how to compile Xtend files. Note that both clients lack any Eclipse configuration, because all the information is present in the build scripts. If you want to look at them inside Eclipse, you need to use their respective Eclipse integration.

For Maven users, there is the xtext-maven-plugin. You just add your core language artifact as a dependency to the plugin and add the language configuration. This configuration contains the name of the StandaloneSetup class and the output folders. If your project has a large classpath, you can also filter which jars to search for model files, greatly speeding up your build. As of now, you manually have to add the output directory to the list of Java source folders in order for the generated code to be picked up by the Java compiler. This is done using the build-helper plugin. Try running the build with mvn clean verify. If you want to see the project in Eclipse. use the "Import existing Maven Project" wizard.

For Gradle users, there is the even more convenient xtext-gradle-plugin. Similar to Maven, you add your core language artifact to the xtextTooling dependency configuration. Then you configure the StandaloneSetup and output folders. You can directly specify that an output folder produces Java code, which will automatically make it a source folder. Try running the build with gradlew clean build. If you want to see the project in Eclipse. run gradle eclipse on the command line and then use the normal "Import existing Project" wizard.

The fully mavenized way

Now for the final part of our journey: Building the language infrastructure from scratch using Maven. You can find the projects under language/2-full. In the parent POM you can see that we have added common configuration for Xtend. This is because part of the language infrastructure and tests are implemented using Xtend. Subprojects now only need to add a two-line configuration to use Xtend and inherit everything else from the parent. The other big change happened in the core language plugin. In order to run the mwe2 workflow to generate the language infrastructure, we use the exec-maven plugin.

We add org.eclipse.xtext.xtext as a dependency, which contains the grammar language and generator including all dependencies like mwe2. We also depend on Xbase because our language inherits from it. Then we just point the Mwe2Launcher at our workflow file. For the proper cleanup, we configure the clean-plugin to throw away everything that is generated by the workflow. By default the clean-plugin only cleans the /target folder. Last but not least there is a one-line change to the workflow itself. The grammarUri cannot be a classpath Uri, since at the time the workflow is run in Maven, the grammar file is not on the classpath. Instead we switch to a platform resource Uri.

Now try running mvn clean to see how everything but the handwritten source gets deleted and then mvn install to build the language from scratch. You could now .gitignore the generated code.

Conclusion

I hope this guide has helped you set up your prefered build style. If you have any further questions, just ask them in the Xtext forum. There are lots of helpful people who will probably have an answer =)