Introduction


Building a domain-specific language (DSL) for structural parts of an application has always been rather easy with Xtext. But structure alone is not sufficient in many cases. When it comes to the behavioral aspects users often fall back to implementing them in Java. The reasons are obvious: expressions and statements are hard to get right and extremely complex and therefore costly to implement.

This document introduces and explains a new API, which allows reusing predefined language constructs such as type references, annotations and fully featured expressions anywhere in your languages. You not only reuse the grammar but the complete implementation including a compiler, interpreter, the whole Java type system and a tight IDE integration. And the best part is, that it is relatively easy to do so. It is actually just two steps:

Write the Grammar

Using the traditional Xtext grammar language you freely describe the syntax of your language. The specialty for JVM languages is, that you inherit from an abstract grammar org.eclipse.xtext.xbase.Xbase, which predefines the syntax for the reusable parts. You do not need to use all of them directly and you can of course change the syntax or add new concepts, as it seems fit.

Map to Java

Having the grammar defined, you now need to tell Xtext what your language concepts mean in terms of Java constructs. For that purpose you use a so-called model inferrer, a special API that allows you to create any number of Java classes, interfaces or members from your DSL. This hook not only defines how your language is translated to Java, but also defines the scope of the embedded expressions. The expressions from your language 'live' in the context you give them. If you want an expression to have access to certain local variables, just put it into a method with appropriate parameters or use instance fields if they work better.

While in plain Xtext you would usually customize a bunch of further services to fit your needs, Xbase already has JVM model aware implementations almost all of them. For example, a generic code generator generates the Java code directly from the inferred model. The same way, Xbase already integrates your language with JDT to an extend that customizations beyond the JVM model inferrer will only be necessary for very special cases. You can naturally mix Java and DSL code in the same application without any barriers.

The inferred model also reveals your DSL constructs to other JVM languages. The Java type system is used as a common hub to integrate arbitrary languages with each other. You can for instance call templates directly from a script and vice versa. You do not even need to generate the equivalent Java code; everything is based on the Java types you create in the model inferrer.

To illustrate the power and flexibility of these two abstractions, we have built seven example languages using them:

Each language is very simple and focuses on the value a DSL can add to the respective viewpoint. It is meant to give you an idea of what is possible without being a complete practical solution. Yet the languages are flexible and come with powerful core abstractions. We also covered different technical aspects of customizing to the language infrastructure. Some languages have special syntax coloring, others provide customized outline views or content assist. All aspects of a language are still fully customizable when using Xbase.

CAUTION: This is Provisional API

Please be aware that some of the new API covered in this document is not yet finalized and will likely be improved in future releases in incompatible ways. Usages of such API are marked with a warning.

Common Requirements


To run any of the examples, you will need Eclipse 3.6 or better for your platform. In addition, you have to install Xtend and of course Xtext 2.7.

If you prefer a simple all-inclusive installation, consider downloading the latest Xtext distribution.

Additional requirements are mentioned in the Running the Example section of each chapter.

Getting the Code

You can get the source code for all languages from the github repository at github.com/xtext-dev/seven-languages-xtext. The repository contains two folders languages and examples. Download the plug-ins from languages into the root workspace and the examples into the runtime workspace spawned from the root one using Run > Run Configurations... > Eclipse Application > Run (<language>).

Each language consists of several Eclipse projects

org.xtext.<language> The base infrastructure
org.xtext.<language>.ui The editor based on Eclipse
org.xtext.<language>.tests Tests for the language
org.xtext.<language>.lib Runtime library
org.xtext.<language>.example Examples for using the language

Some of the languages do not include all of these plug-ins but the general structure is always the same.

A Short Xtend Primer


Any general code in the examples is implemented in Xtend. Xtend is a more expressive and less verbose way to implement Java applications. It is 100% interoperable with Java APIs and compiles to readable Java code. In addition, it uses the same expressions that we use in our example languages. In fact it is built with the very same API that is described in this document. This should give you a taste of how powerful JVM-languages built with Xtext can actually be.

Xtend is designed to be easy to learn for Java developers. In this section we will shortly describe the most important language features that were used in our examples. For a full description of the Xtend language, please refer to the Xtend documentation.

Just like a Java file, an Xtend file starts with a package declaration and an import section followed by one or more class declarations. Semicolons are optional. All types are public by default. Xtend classes can extend other classes and implement interfaces just like in Java. It does not make any difference whether they were originally declared in Xtend or in Java.

JavaBean properties can be accessed directly by their name. The access will be automatically mapped to the appropriate accessor method. That means you can write

println(foo.bar)      // instead of println(foo.getBar())
foo.bar = baz         // instead of foo.setBar(baz)
foo.fooBars += foobar // instead of foo.getFooBars().add(foobar)

Empty parentheses on method or constructor calls can be skipped.

Methods are introduced with the keyword def or override if they override/implement a super type's method. Methods are public if not specified otherwise. The value of the last expression is returned if no explicit return expression is used and the method's return type is not void.

Variables are declared with the keywords val (final) or var (non-final). Field declarations can use the same syntax as in Java.

Xtend is statically typed, but you do not have to specify the type in a declaration if it can be inferred from the context:

val x = newArrayList('foo', 'bar', 'baz') // x is of type List<String>
def foo() {  // equivalent to  def int foo()...
  1
}

The methods of fields marked as extension are callable in extension syntax. That means, you can put the first argument in front as if it were the receiver of the call, e.g.

// assume the class Foo defines a method foo(Baz)
extension Foo theFoo

def bar(Baz baz) {
  baz.foo  // calls theFoo.foo(baz)
}

Static methods can be put on the extension scope with a static extension import, e.g.

import static extension java.util.Collections.*
...
val foo = singleton('foo') // calls Collections.<String>singleton('foo')

In addition to Java's this, you can define another implicit receiver variable named it. As with this, you can omit it in feature calls, i.e.

class Foo {
  def foo(Bar it) {
    foo // will call it.foo() or if that doesn't exist this.foo()
  }
}

Xtend provides lambda expressions. These are anonymous functions in square brackets.

[String foo, String bar | foo + bar]  
  // a function (String foo, String bar) { foo + bar }

As this is a bit bulky, there are more rules to make working with lambdas more attractive:

  1. When a lambda expression is the last argument in a method call, it can be put behind the closing parentheses.
  2. Lambdas are automatically coerced to interfaces with a single function. Parameter types will be inferred.
  3. If you skip the declaration of the only parameter, it will be implicitly called it.

new Thread [ println("Hello concurrent world") ] 
  // lambda will be coerced to a java.lang.Runnable
val list = #['fooooo', 'fo', 'foo']  // #[] delimits a list literal
list.sortBy[ length ]   
  // lambda is coerced to a function (String)=>Comparable
  // equivalent to list.sortBy[String it | it.length]

Also noteworthy is the template expression, which allows defining interpolated multi-line string literals. In addition, the template expressions has intelligent whitespace handling and supports a special FOR and IF construct, which is far better readable in this context. In model inferrers, the template expressions are enhanced further to support type literals, automatically adding an import to the generated Java file.

We most often use this expression in the examples to generate some synthetic Java boilerplate code. Here is an example from the http routing language:

'''
  String url =  request.getRequestURL().toString();
  «FOR route : routes»
    {
      //java.util.regex.Matcher will be imported in the generated Java file
      «Matcher» _matcher = _pattern«route.index».matcher(url);
      if (_matcher.find()) {
        «FOR variable : route.url.variables»
            String «variable.name» = _matcher.group(«variable.index + 1»);
        «ENDFOR»
        «IF route.condition != null»
          if («route.nameOfRouteMethod»Condition(request, response
            «FOR v : route.url.variables 
             BEFORE ", " 
             SEPARATOR ", "»«v.name»«ENDFOR»))
        «ENDIF»
        «route.nameOfRouteMethod»(request, response
          «FOR v : route.url.variables», «v.name»«ENDFOR»);
      }
    }
  «ENDFOR»
'''

Scripting Language


The scripting language allows writing code without any preludes such as package, class or method declarations. Just open a file and start coding. It's the simplest of the seven languages and is therefore a good starting point.

Overview

As you can see the language is straightforward: All you can do is write expressions. The expressions you see are the same as in Xtend and they are defined in its own grammar (called Xbase). They are syntactically very close to Java, but feature advanced concepts like lambda expressions and operator overloading. They support local type inference so you do not have to write types everywhere like in Java, but the expressions are still statically typed. For a Java developer, the code should be self-explanatory. The Xbase expression language is exhaustively covered in the Xtext documentation.

A script is compiled to a Java class with one main method. That is the script

println('Hello World!')

is compiled to the following Java source code

package my.first;

public class Application {
  public static void main(String... args) {
    System.out.println("Hello World!");
  }
}

Note that the core language infrastructure such as the parser, linker and compiler does not depend on Eclipse, but the compilation is integrated with Eclipse for convenience reasons. All 7 languages can be parsed, compiled and executed without Eclipse. We could also make use of the interpreter, which might make more sense for a scripting language. How to use and integrate the interpreter is explained in the section describing the language.

Running the Example

Make sure you have the projects org.xtext.scripting and org.xtext.scripting.ui in your workspace. Then start a new Eclipse by choosing Run > Run Configurations... > Eclipse Application > Run (org.xtext.scripting). Import the project org.xtext.scripting.examples into the newly spawned workspace using the Import existing projects into workspace wizard.

Grammar

To build a language with Xtext, you first have to define a grammar. As opposed to other parser generators, an Xtext grammar defines both, the lexical structure of the language and an object model (the AST or semantic model) that is build during parsing. For a more detailed description of the Xtext grammar language, please see the Xtext documentation.

The grammar for our DSL is rather simple. We inherit from org.eclipse.xtext.xbase.Xbase to get the syntax of the expressions. As we want to refer to the type XBlockExpression (src), we have to import Xbase's Ecore model. The single type inferred from this grammar goes into the Ecore model simpleExpressions.

grammar org.xtext.scripting.Scripting with org.eclipse.xtext.xbase.Xbase

generate scripting "http://www.xtext.org/scripting/Scripting"
import "http://www.eclipse.org/xtext/xbase/Xbase"

Script returns XBlockExpression:
	{Script}
	(expressions+=XExpressionInsideBlock ';'?)*;

The main rule Script is defined to produce an object of type Script, which is a subtype of XBlockExpression. A block expression simply contains any number of expressions. The rule XExpressionInsideBlock is defined in the Xbase grammar. Usually block expressions are surrounded by curly braces, but of course we do not want to force anybody to write curly braces at the beginning and the end of a simple script.

Translation to Java

To make our language executable, we have to define how its concepts relate to Java concepts. In Xtext, this is defined by the IJvmModelInferrer (src). The language generator automatically generates an Xtend stub for it. Nevertheless, it is up to the language developer to implement the infer() method.

This hook is not only used to explain how to generate Java code, but also to give expressions a proper scope and to make your DSL constructs visible by other JVM languages. The Java type system is used as a common hub to integrate arbitrary languages with each other.

The inferrer is written in Xtend, if you are not yet familiar with it, you should read at least the Xtend Primer first.

The JVM model inferrer code for the scripting language looks like this:

class ScriptingJvmModelInferrer extends AbstractModelInferrer {

  @Inject extension JvmTypesBuilder

  def dispatch void infer(Script script, 
                          IJvmDeclaredTypeAcceptor acceptor, 
                          boolean isPreIndexingPhase) {
    val className = script.eResource.URI.trimFileExtension.lastSegment
   	acceptor.accept(script.toClass(className))[
      // the class gets one main method
      members += script.toMethod('main', typeRef(Void.TYPE)) [
   	    parameters += script.toParameter("args",  typeRef(String).addArrayTypeDimension)
   	    static = true
        varArgs = true
        // Associate the script as the body of the main method
        body = script
      ]	
    ]
  }
}

It maps each script to one Java class with a main method. Note the use of script as a receiver for all the factory methods. When creating a Java element, you need to pass a context so Xtext knows what the origin of a Java element is. This is used throughout the workbench for features like find references, call hierarchies, rename refactoring and more.

As the whole script is an expression it is associated with the body of the main method. That association is important since it defines the scope of the expression. It includes the variables, fields and methods that are visible as well as which return type is expected.

In this case void is expected, so you would get an error when using a return expression. Also the parameter args defined in the main method is now on the scope. So you could write the following script:

for (arg : args)
  println('-- '+arg)

As you can imagine, this is a fairly powerful way to put things on the scope implicitly.

Build Language


Build tools like Ant or Gradle decompose the build process into a set of tasks. A task can stand for a compilation step, copying some files, bundling, running tests etc. The order of execution is calculated from the dependencies of the defined tasks.

This build language combines the declarative approach of defining tasks and dependencies with the full power of a modern expression language. It is a bit similar to Ant but statically typed and with very good IDE support.

Overview

The above screenshot shows an exemplary build script. A script has a couple of parameters. The tasks define dependencies to other tasks. What happens if the task is executed is defined in a block within curly braces.

A build script can be run from the command line, assigning values to the parameters in the syntax myscript --<paramName> <value>.

We have put most functionality of the language in the runtime library: Detecting the order of execution, parsing parameters, etc. The language itself focusses on the structural parts, and leaves the actions to Xbase. This allows both us and future users to add new tasks as library methods instead of hard-coding them in the language.

Running the Example

In the runtime workspace, open the BuildExample.build in the editor. The example project comes with some sample java code to compile in the example-project folder. Choose Run as > Build Task from the context menu of any task to execute it.

Grammar

The grammar of the DSL is once again quite slim:

grammar org.xtext.builddsl.BuildDSL with org.eclipse.xtext.xbase.Xbase

generate build "http://www.xtext.org/builddsl"

BuildFile:
	("package" name=QualifiedName)?
	importSection=XImportSection?
	declarations+=Declaration*;

Declaration:
	Task | Parameter;

Parameter:
	'param' type=JvmTypeReference? name=ValidID ('=' init=XExpression)?;

Task:
	'task' name=ValidID 
	('depends' depends+=[Task|ValidID] (',' depends+=[Task|ValidID])*)?
	action=XBlockExpression;

A BuildFile starts with a package declaration. The generated Java class will be located in this namespace. The next part is an importSection. Since version 2.4, Xbase includes extensive tooling to validate and organize import statements. To make this available in your language, you just have to include an XImportSection as in this example. The imports are followed by the Declarations. A Declaration can be a Task or a Parameter. A Parameter can declare a type and an initialization expression. Tasks define dependencies on other tasks by means of an Xtext cross-reference. They also contain an action, which is a XBlockExpression (src) from Xbase, thus everthing is possible within a task.

Translation to Java

For each BuildFile we create a Java class that extends the library class BuildScript (src). We generate a main method allowing to execute the script as a Java application with command line parameters. The use of System.exit allows to return error codes to the caller.

DSL

package sample

Java

package sample;
...
public class SimpleBuild extends BuildScript {
...
  public static void main(final String... args) {
    SimpleBuild script = new SimpleBuild();
    if (script.showHelp(args)) {
      System.exit(HELP);
    }
    System.exit(script.doBuild(args));
  }
...
}

The respective inferrer code looks like this:

class BuildDSLJvmModelInferrer extends AbstractModelInferrer {

  @Inject extension JvmTypesBuilder

  def dispatch void infer(BuildFile file, 
                          extension IJvmDeclaredTypeAcceptor acceptor, 
                          boolean isPreIndexingPhase) {
    val qualifiedName = file.javaClassName
    val simpleName = Strings.lastToken(qualifiedName, ".")
    accept(file.toClass(fqn))[
      superTypes += typeRef(BuildScript)
...      
      val stringArray = typeRef(String).addArrayTypeDimension
      members += file.toMethod("main", typeRef(void)) [
        parameters += file.toParameter("args", stringArray)
        varArgs = true
        static = true
        body = '''
          «scriptName» script = new «scriptName»();
          if (script.showHelp(args)) {
            System.exit(HELP);
          }
          System.exit(script.doBuild(args));
        '''
      ]
    ]
...

Each Task becomes a method in the Java class. A @DependsOn (src) annotation communicates the dependencies to the runtime. The superclass will scan for such annotations and execute the dependencies in the right order.

DSL

task print depends prepare {
  print(name)
}

Java

@DependsOn("prepare")
protected void print() {
  InputOutput.<String>print(this.name);
}

The annotation part may be interesting, so here is the snippet from the inferrer:

// a method for the actual task body
members += file.tasks.map[ task | toMethod(task.methodName, typeRef(Void.TYPE)) [
  visibility = JvmVisibility.PROTECTED
  annotations += annotationRef(DependsOn, task.depends.map[name])
  body = task.action
  ]
]

Finally, we create a field with the @Param (src) annotation from each Parameter. The superclass will make the so marked fields initializable from command line arguments.

DSL

param name = 'World'

Java

@Param
public String name = "World";

The type can be skipped. If there is an initialization expression, the parameter's type is inferred from the initialization expression. If that one is missing, too, String is assumed. The Elvis-operator ?: comes handy for this use case. The different sources are tried to find the best type for the field. In the inferrer, this looks like:

@Inject ITypeProvider typeProvider
...
val type = declaredParameter.type 
	?: declaredParameter?.init?.inferredType
	?: typeRef(String)

Validation

When Tasks are depending on each other, cycles will break the computation of the execution order. There is a check for this constraint in the validator BuildDSLValidator (src):

class BuildDSLValidator extends XbaseJavaValidator {
...
  @Check
  def void checkNoRecursiveDependencies(Task task) {
    task.findDependentTasks [ cycle |
      if (cycle.size == 1) {
        error('''The task '«task.name»' cannot depend on itself.''', 
            cycle.head, DECLARATION__NAME, CYCLIC_DEPENDENCY)
      } else {
        error('''There is a cyclic dependency that involves tasks «
                cycle.map[name].join(", ")»''', 
              cycle.head, DECLARATION__NAME, CYCLIC_DEPENDENCY)
      }
    ]
  }
...

Imports

By using the XImportSection form Xbase, the language automatically supports the notion of plain imports, static imports and static extension imports. While the first two work as in Java, a static extension import puts the static methods of the specified calls on the extension scope, such that it can be called as if it were a method on the first argument. See the Xtend primer for a more detailed description.

We ship some predefined extension classes to enhance the Java classes File and Class. These are always put onto the extension scope by a customized BuildDSLImplicitlyImportedTypes (src):

class BuildDSLImplicitlyImportedTypes extends ImplicitlyImportedFeatures {
	/**
	 * Add methods from {@link FileExtensions} and {@link ClassExtensions} to the extension scope.
	 */
	override protected getExtensionClasses() {
		(super.getExtensionClasses() + #[FileExtensions, ClassExtensions])
			.toList
	}
}

As always, such an implementation has to be bound in the BuildDSLRuntimeModule (src):

public class BuildDSLRuntimeModule 
    extends org.xtext.builddsl.AbstractBuildDSLRuntimeModule {
...
  public Class<? extends ImplicitlyImportedFeatures> 
      bindImplicitlyImportedFeatures() {
    return BuildDSLImplicitlyImportedTypes.class;
  }
}

Operator Overloading

As we expect a build language to deal with files and directories a lot, we have extended the syntax around these in the FileExtensions (src). We leverage the fact that Xbase allows to overload operators by means of library extensions. The following examples show how to avoid the noisy constructor calls to File:

DSL

val dir = 'someDir'.file
val file = dir / 'fileName.txt'
val theSameFile = 'someDir' / 'fileName.txt'

Java

final File dir = FileExtensions.file("someDir");
final File file = FileExtensions.operator_divide(dir, "fileName.txt");
final File theSameFile = FileExtensions.operator_divide(
    "someDir", 
    "fileName.txt");

The compiler replaces operators with method calls to methods named operator_<operatorName>(). These have to be callable on the left operand, i.e. be a method of the left operands type or be added as an extension to the left operand type. In this case, we use the latter approach, since File as well as String are sealed types. To learn more about operators and operator overloading please consult the Xtext documentation.

Run as... Integration

To facilitate executing build scripts from within Java, we have extended Eclipse's launch support. The respective classes are BuildDSLLaunchDelegate (src), BuildDSLLaunchShortcut (src) and BuildDSLLaunchTabGroup (src). Describing all the details would be a bit lengthy but it is quite straightforward if you are familiar with the launching API.

DSL for MongoDB


MongoDB is a very popular document-based database management system. In mongoDB, database entries (AKA documents) have fields, which are essentially (key, value) pairs. MongoDB is schema free, i.e. there are no rules, which fields have to be defined and of what type they are. This allows for very flexible and heterogeneous data structures and is a perfect match with JSON.

OTOH, Java is statically typed: The available types, their field names and field types are known and validated at compile time. The JavaBeans convention defines how instances can be manipulated. The standard mongoDB Java driver reflects the fact that mongoDB is schema-free by providing mongo documents as plain Java maps.

With this language you can describe statically typed Java-facades for MongoDB documents without hiding the dynamic nature of them. The language uses a tree-like syntax similar to JSON but lets you add static Java type information.

Overview

In this project, we have created a small DSL mongoBeans based on Xtext that allows to create basic entity classes. These are backed by mongoDB objects but provide a statically typed JavaBeans API. Think of the language as a description how to map mongoDB documents to JavaBeans, in analogy to well known object relational mappers.

An example mongoBeans file looks like this:

import java.util.*

package org.musicdb {
    // a mongo bean
    Artist {
        String name    // single valued property
        Album* albums  // array property
        // an operation
        Iterable<Track> getOeuvre() {
            albums.map[track].flatten
        }
    }

    // another mongo bean
    Album {
        String title 
        int year 
        // inline definition of a mongo bean
        Track {        
            String title
            int seconds
        }* tracks
    }
}

For each MongoBean definition in a MongoFile file, we generate a Java class that wraps a DBObject. The class provides statically typed getter and setter methods for all defined MongoProperties. In the implementation of these accessor methods we delegate to the wrapped DBObject and do all the casting and conversion work. For the Artist in the above example, this would look like

public class Artist implements IMongoBean {

  private DBObject _dbObject;  
  ...
  public String getName() {
    return (String) _dbObject.get("name");
  }
  
  public void setName(final String name) {
     _dbObject.put("name", name);
  }
  ...
}

By using the generated Java code, the rest of the application can use a type-safe and JavaBeans conformant API to access the data model. In addition, MongoBeans can define MongoOperations, which are translated to Java methods. We can use MongoProperties as well as Java types inside the operations' bodies.

Client code could then look like this:

Artist john = new Artist();
john.setName("John Coltrane");
Album album = new Album();
album.setTitle("A Love Supreme");
john.getAlbums().add(album);
Track...  // create some tracks and add them to the album
 
System.out.println(john.getName() + "'s Oeuvre");
for(Track track: john.getOeuvre()) 
  System.out.println(track.getTitle());

DBCollection dbCollection = ... // standard mongoDB driver code
dbCollection.save(john.getDBObject())

Running the Example

In addition to the common requirements, you need the mongoDB implementation for your platform. We have included the mongoDB Java driver from Eclipse Orbit in the code base.

Import the projects into an Eclipse workspace and run the launch configuration Run (org.eclipse.xtext.mongobeans). Import the example plug-in into the new workspace and run MusicDBXtendTest as a JUnit test.

Grammar

The complete mongoBeans grammar looks like this:

grammar org.xtext.mongobeans.MongoBeans with org.eclipse.xtext.xbase.Xbase

generate mongoBeans "http://www.eclipse.org/xtext/mongobeans/MongoBeans"

MongoFile:
	importSection=XImportSection?
	elements+=AbstractElement*;

AbstractElement:
	PackageDeclaration | MongoBean;

PackageDeclaration:
	'package' name=QualifiedName '{'
		elements+=AbstractElement*
	'}';

MongoBean:
	name=ValidID '{'
		features+=AbstractFeature*
	'}';

AbstractFeature:
	MongoOperation | MongoProperty;

MongoProperty:
	(type=JvmTypeReference | inlineType=MongoBean) (many?='*')? name=ValidID;

MongoOperation:
	=>(returnType=JvmTypeReference name=ValidID '(') 
		(parameters+=FullJvmFormalParameter 
			(',' parameters+=FullJvmFormalParameter)*
		)? 
	')' 
	body=XBlockExpression;

The language inherits from the Xbase grammar in order to allow Xbase expressions and references to Java elements. A MongoFile starts with an import section (see Build DSL for details). The import section is followed by any number of AbstractElements, which can be PackageDeclarations or MongoBeans. Note that as opposed to Java, PackageDeclarations can be nested. MongoBeans define statically typed MongoProperties, which can be single-valued or multi-valued denoted by an * following the type name. The type of a MongoProperty can also be defined inline. MongoBeans can also define MongoOperations. The body of such an operation is an XBlockExpression from Xbase.

Translation to Java

The JVM model inference is implemented in the MongoBeansJvmModelInferrer (src). As the generated code is quite rich, this is the most complex component of this language.

For each MongoBean, we create a Java class implementing the interface IMongoBean (src). This interface is the first type of a small runtime library that has to be on the classpath at runtime.

DSL

package org.musicdb {
  Artist { ...

Java

package org.musicdb;
...
public class Artist implements IMongoBean {  ...

The inferrer code responsible for this section looks like this:

@Inject extension JvmTypesBuilder
@Inject extension IQualifiedNameProvider  
...
def dispatch void infer(MongoFile file, 
                        IJvmDeclaredTypeAcceptor acceptor, 
                        boolean isPreIndexingPhase) {
  for(bean : file.eAllOfType(MongoBean)) {
       acceptor.accept(bean.toClass(bean.fullyQualifiedName))

documentation = bean.documentation superTypes += typeRef(IMongoBean) ... // calling various methods to create Java members // from the AbstractFeatures ] } }

DSL for Guice


The Guice DSL is a little language that lets you define Guice modules in a readable and declarative way.

Overview

Guice is a great dependency injection container, which uses Java types and annotations to declare and refer to injection points. You could for instance have the following field:

@Inject
@Named("properties") Map<String,String> properties;

A module is used to tell the framework what instance to inject into such injection points. For that, Guice comes with a fluent interface API written in Java and a couple of tricks (e.g. TypeLiteral) to make the configuration as readable and maintainable as possible.

You could for instance declare the following module:

public class MyModule implements Module {
  @Override
  public void configure(Binder binder) {
    Map<String,String> properties = Maps.newHashMap();
    properties.put("debugLevel", "info");
    
    binder.bind(new TypeLiteral<Map<String,String>>(){})
      .annotatedWith(Names.named("properties"))
      .toInstance(properties);
  }
}

The big advantage of using Java over an external text or XML file is that you can leverage the IDE and the type checking. We want to have that. The downside is that you have to trick (i.e. TypeLiteral) a lot in order to have an agreeable syntax. Also since the configuration is 'hidden' in a method implementation and not really declarative you cannot validate a Guice module at compile time.

The Guice DSL described in this section lets you describe the module above like this:

MyModule {
  @Named("properties") Map<String,String> 
    to-instance newHashMap('debugLevel' -> 'info')
}

This not only uses the exact same syntax one uses in any injection points, but also opens up all kinds of possibilities for static analysis. Usually the instantiation of a Guice injector at runtime takes quite some time, because then all the very helpful validation is done. A language like the one described in this section could do all theses analysis at compile time, that way speeding up start up of the whole application significantly.

Running the Example

In the example located in the project org.xtext.guicemodules.examples two modules are declared, one for a possible runtime scenario and one for a test scenario (yes, you sometimes want a module for tests).

import com.acme.*
import com.acme.impl.*

com.acme.RuntimeModule {
  bind DataProvider to FileDataProvider
  bind @DataFile String to-instance 'my-data.txt'
}

com.acme.TestModule mixin RuntimeModule {
  bind DataProvider to-instance ['dummy-data' ]
  bind LoggingService to BufferedLoggingService
}

You can see the two modules in action by running com.acme.Main from the context menu as a Java application or as a JUnit test.

Grammar

The grammar is less than 30 lines long. It allows declaring any number of imports using the import mechanism already described for the scripting language. A module can 'mixin' any number of other modules, which allows to reuse existing modules but override keys with different bindings. The mixin feature is described as a cross reference to another ModuleAST. Cross references are covered in detail in the Xtext documentation.

The language allows binding keys to other keys and to instances. Other concepts like binding to providers is something you could do, but would not help in getting the idea across. Everybody is encouraged to fork this first prototype and build something really useful ;-)

Instead of extending org.eclipse.xtext.xbase.Xbase the grammar org.eclipse.xtext.xbase.annotations.XbaseWithAnnotations is extended which adds full support for annotations. You just have to refer to the rule XAnnotation as it is done in the rule KeyAST. Btw. it is sometimes a good idea to suffix (or prefix) the AST node types to avoid confusion when working with a library where concepts are named similarly.

grammar org.xtext.guicemodules.GuiceModules 
   with org.eclipse.xtext.xbase.annotations.XbaseWithAnnotations
 
import "http://www.eclipse.org/xtext/common/JavaVMTypes" as types
generate guiceModules "http://www.xtext.org/guicemodules/GuiceModules"
  
ModulesAST :
	importSection=XImportSection?
	modules+=ModuleAST*;

ModuleAST :
	name=QualifiedName ('mixin' mixins+=[ModuleAST|QualifiedName] 
	  (',' mixins+=[ModuleAST|QualifiedName])*)? 
	'{'
		bindings+=BindingAST*
	'}'
;

BindingAST:
	'bind' from=KeyAST 
	('to' to=KeyAST | 'to-instance' toInstance=XExpression)?;

KeyAST:
	annotation=XAnnotation? type=JvmTypeReference;

Translation to Java

A module is mapped to a single java class. The 'mixin' modules are not translated to Java inheritance but to a delegation approach. Here is how a simple module declaration with a single mixed-in module is translated.

DSL

MyModule mixin OtherModule { 
}

Java

public class MyModule implements Module {

  private OtherModule otherModule = new OtherModule();

  public void configure(final Binder binder) {
    configure(binder, new HashSet<com.google.inject.Key<?>>());
  }

  public void configure(final Binder bind, 
      final Set<Key<? extends Object>> usedKeys) {
    try {
      testModule.configure(bind, usedKeys);
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }
}

As you can see, the language simply leverages the nice equals/hashCode implementation of Guice's Key to implement the override semantics of mixed-in modules.

The corresponding parts in GuiceModulesJvmModelInferrer (src) are:

accept(module.toClass(module.fullyQualifiedName))[
...      
  // declare a field for each mixed-in module
  for (mixin : module.mixins) {
    if (!mixin.eIsProxy)
      members += mixin.toField( mixin.simpleName, 
        typeRef(mixin.fullyQualifiedName.toString)) [
        initializer = '''new «mixin.name»()'''
      ]
  }
  
  // and later when declaring the configure method
  members+= module.toMethod("configure", typeRef(void)) [
    documentation = 'Registers bindings for keys not present in ...'
    parameters += module.toParameter("bind", typeRef(Binder))
    parameters += module.toParameter("usedKeys", typeRef(Set, typeRef(Key, wildcard)))
    body = '''
      try {
        ...some other code
        «FOR mix : module.mixins»
          «mix.simpleName».configure(bind, usedKeys);
        «ENDFOR»
      } catch (Exception e) {
        throw new RuntimeException(e);
      }
    '''
  ]

Expressions are used in to-instance bindings and as always they need a proper scope to live in. The simplest way is to declare a private method for each expression.

for (binding : module.bindings) {
  // if it's a toInstance binding, create a synthetic
  // method to give the expression a proper scope
  if (binding.toInstance != null) {
    members += binding.toMethod(binding.syntheticToInstanceName, 
        binding.from.type) [
      visibility = JvmVisibility.PRIVATE
      body = binding.toInstance
    ]
  } 
...

In order to obtain instances of annotations as required by Guice's Binder, we declare dummy fields for annotated types and use reflection to get corresponding instances.

DSL

com.acme.RuntimeModule {
  bind @DataFile String to-instance 'my-data.txt'
}

Java

// declaration of dummy field with annotation
@DataFile
private String _from0;
  
// and later the following code is used to get the key
  Key<java.lang.String> key = 
      Key.get(new TypeLiteral<String>(){}, 
          getClass().getDeclaredField("_from0").getAnnotations()[0]);

The two sections in the model inferrer responsible for this are:

for (binding : module.bindings) {
...
  // if a key has an annotation, declare a field so we can use that 
  // annotation via reflection later.
  if (binding.to?.annotation != null) {
    members += binding.toField(binding.to.syntheticName, binding.to.type) [
      addAnnotation(binding.to.annotation)
      visibility = JvmVisibility.PRIVATE
    ]
  }
  if (binding.from.annotation != null) {
    members += binding.toField(binding.from.syntheticName, 
        binding.from.type) [
      addAnnotation(binding.from.annotation)
      visibility = JvmVisibility.PRIVATE
    ]
  }
}

// and the following method
def guiceKey(KeyAST it) '''
  Key.get(new TypeLiteral<«type>(){}«
  IF annotation != null
  », getClass().getDeclaredField("«syntheticName»").getAnnotations()[0]«
  ENDIF»)'''

That is basically it. The rest should hopefully be self-explanatory.

Validation

One of the sweet spots for a Guice modules DSL is the ability to do a lot of the validation usually done at runtime during compile time. Since this is just an example it just scratches the surface. There is just a single compiler check validating whether any used annotation is itself annotated with BindingAnnotation.

This is the relevant code from GuiceModulesValidator (src):

@Check def checkAnnotationIsBindingAnnotation(XAnnotation it) {
  switch type : annotationType {
    JvmAnnotationType: 
      if(!type.annotations.exists[ 
          annotation.is(BindingAnnotation) 
         ])
		error("The annotation is not annotated with @BindingAnnotation", 
		XANNOTATION__ANNOTATION_TYPE)
  }
}

It would be really cool to leverage the full information, which is available and analyze the dependencies transitively so you get feedback while you type for any unfulfilled dependencies. Also using all the information to compile a ready to use Injector instead of a module seems interesting.

Http Routing Language


This is a little language that lets you dispatch incoming HTTP requests.

Overview

For server-side web development one needs to match any incoming HTTP requests to some handling code. A request is defined by a HTTP method (i.e. GET, POST, etc.), a URL and maybe some other data (headers, parameters, etc.). In the Java world there are many different solutions to this problem. Java Servlets come with an external configuration (web.xml), where you configure what servlet should be called for which URLs. JAX-RS (JSR-311) uses annotations to match incoming URLs and other frameworks have their own external DSLs for that.

The HTTP routing language explained in this section uses an approach very similar to the one from the Play framework. You basically declare a list of URL patterns and explain what to do for each case. In contrast to Play! which heavily relies on static methods, you can also declare so called dependencies which are translated to fields annotated with @Inject. So this DSL plays nicely with dependency injection and especially with the Guice modules DSL.

inject GuessTheNumber controller

GET /guess/:theGuess
  do controller.handleGuess(theGuess)

As you can see, you can have named variable placeholders in the URL and use them in the do-part. There also is a when-part which allows to specify an additional condition using the request object as well as any parameters:

inject GuessTheNumber controller

GET /guess/:theGuess
  when !controller.isValidGuess(theGuess)
  do controller.handleWrongRange(theGuess)

GET /guess/:theGuess
  do controller.handleGuess(theGuess)

Running the Example

The example project's name is org.xtext.httprouting.examples and includes a simple but runnable number guessing game. Just start the server (an embedded Jetty) by running framework.StartServer as a Java Application. Then point your browser to the URL http://localhost:8080/guess.

Grammar

The Routing DSL extends org.eclipse.xtext.xbase.annotations.XbaseWithAnnotations to make use of expressions predefined by Xbase plus support for Annotations.

grammar org.xtext.httprouting.Route 
  with org.eclipse.xtext.xbase.annotations.XbaseWithAnnotations

generate route "http://www.xtext.org/httprouting/Route"
import "http://www.eclipse.org/xtext/common/JavaVMTypes" as types
import "http://www.eclipse.org/xtext/xbase/Xbase" as xbase

Model :
	importSection=XImportSection?
	declarations+=AbstractDeclaration*;

AbstractDeclaration :
	Dependency | Route;

Dependency :
	'inject' annotations+=XAnnotation? type=JvmTypeReference name=ID;

Route :
	requestType=RequestType url=URL 
	('when' condition=XExpression)? 
	'do' call=XExpression;

enum RequestType :
	GET | POST | PUT | DELETE | HEAD;

/**
 * matches URLs like 
 * 	'/foo/bar.html' or 
 * 	'/customer/:customerID/save'
 */
URL :
	{URL} 
	( '/' | ('/' (QualifiedName | variables+=Variable))* 
	('/' variables+=Variable wildcard?='*')?);

Variable :
	':' name=ID;

There should not be any surprises if you know the grammar language: A Model consists of an XImportSection followed by any number of Dependencies and Routes. A Route starts with a RequestType, i.e. HTTP method, followed by a URL pattern. Then an optional when-clause can be specified followed by a mandatory do-clause.

Translation to Java

In RouteJvmModelInferrer (src) you can see that a Java class extending the class HttpServlet is derived. First the Dependencies are translated to Java fields. This is almost a one-to-one mapping.

// translate the dependencies to fields annotated with @Inject
for (field : model.declarations.filter(Dependency)) {
  members += field.toField(field.name, field.type) [
    annotations += annotationRef(Inject)
		addAnnotations(field.annotations)
  ]
}

Next up a field for the URL patterns is generated and a method for the used expressions, such giving them a proper scope and context.

// declare fields for the URL regexp, a method for each when-part 
// and of course the call part of a route
for (route : model.routes.filter[ url != null ]) {
  members += route.toRoutePatternField
  if (route.condition != null)
    members += route.toRouteConditionMethod
  members += route.toRouteCallMethod
}

Note that the code in a model inferrer has to be very defensive, because it is called for any kind of broken models. You just cannot assume that the URL is set although it is mandatory in the grammar, because the user might have written syntactically incorrect code.

Next up the handler methods from HttpServlet are implemented such that they dispatch according the URL patterns and when-clauses. Here is a translated example:

DSL

import com.acme.GuessTheNumber

inject GuessTheNumber controller

GET /guess/:theGuess
  do controller.handleGuess(theGuess)

Java

@SuppressWarnings("serial")
public class NumberGuessing extends HttpServlet {
  @Inject
  private GuessTheNumber controller;
  
  private static Pattern _pattern2 = Pattern.compile("/guess/(\\w+)");
  
  public void _doGet2(final HttpServletRequest request, 
                      final HttpServletResponse response, 
                      final String theGuess) {
    this.controller.handleGuess(theGuess);
  }
  
  @Override
  public void doGet(final HttpServletRequest request, 
                    final HttpServletResponse response) {
    String url =  request.getRequestURL().toString();
    {
      Matcher _matcher = _pattern2.matcher(url);
      if (_matcher.find()) {
        String theGuess = _matcher.group(1);
        _doGet2(request, response, theGuess);
      }
    }
  }
}

As you can see the expression controller.handleGuess(theGuess) is put into a method with three parameters. This is done in the following method from RouteJvmModelInferrer (src):

/**
 * Creates a method for the route's target call.
 * Gives scope and live to the expression.
 */
def protected toRouteCallMethod(Route route) {
  route.toMethod(route.nameOfRouteMethod, typeRef(Void.TYPE)) [
    parameters += route.toParameter("request",  typeRef(HTTP_REQUEST))
    parameters += route.toParameter("response", typeRef(HTTP_RESPONSE))
    for (variable : route.url.variables) {
      parameters += variable.toParameter(variable.name, typeRef(String))
    }
    body = route.call
  ]
}

Just because of that code you can now refer to the local variables request, response, and theGuess. Also it defines that the expected type is void so you are not allowed to write thing like return 42.

Template Language


This is a little template language specialized in generating HTML documents.

The language allows web designers to do their job and lets developers put in the dynamic parts. The syntax and terminals are chosen to be readable and allow rendering the templates in the browser as well as in any HTML 5 compatible WYSIWYG editors. Still, when opened in the DSL editor you get the fully featured, statically typed Eclipse editor.

Overview

A template language works in two modes: Plain text mode, where everything goes directly into the output and the expression mode, where expressions have to be evaluated and the result is inserted into the text. To switch between text mode and expression mode, we use the French quotes « and ». A document starts in text mode.

The template will be compiled to a Java class with a generate(params) method. You can provide additional information like a package declaration, imports and parameters in the preamble inside a template at the beginning of the document.

We provide additional FOR-ENDFOR and IF-ELSE-ENDIF statements to iterate / branch over fixed text blocks. To distinguish them from the Xbase expressions with the same names, they are in uppercase.

Running the Example

In the runtime workspace, run the GenerateHtml file as a Java application (Run as... > Java Application from the context menu). This will execute the template MyWebsite and print the result to the console.

Grammar

This is the grammar of the templates DSL:

grammar org.xtext.template.Template 
   with org.eclipse.xtext.xbase.annotations.XbaseWithAnnotations

generate template "http://www.xtext.org/template/Template"
import 'http://www.eclipse.org/xtext/xbase/Xbase' as xbase

TemplateFile:
	'<!--''«' 
		('package' package=QualifiedName)? 
		importSection=XImportSection? 
		params+=Parameter* 
		body=RichString;

Parameter:
	annotations+=XAnnotation* 
	'param' type=JvmTypeReference? name=ID ('=' defaultexp=XExpression)?;

RichString returns xbase::XBlockExpression:
	{RichString}  
	expressions+=RichStringLiteral 
	(expressions+=RichStringPart expressions+=RichStringLiteral)*;

RichStringLiteral returns xbase::XStringLiteral:
	{RichStringLiteral} value=TEXT;

RichStringPart returns xbase::XExpression:
	XExpressionInsideBlock |
	RichStringForLoop |
	RichStringIf;

RichStringForLoop returns xbase::XForLoopExpression:
	{RichStringForLoop}
	"FOR" declaredParam=JvmFormalParameter ':' forExpression=XExpression
		eachExpression=RichString
	"ENDFOR";

RichStringIf returns xbase::XIfExpression:
	{RichStringIf}
	"IF" if=XExpression
		then=RichString
	(else=RichStringElseIf | "ELSE" else=RichString)?
	"ENDIF";
	
RichStringElseIf returns xbase::XIfExpression:
	{RichStringIf}
	"ELSEIF"if=XExpression
		then=RichString
	(else=RichStringElseIf | "ELSE" else=RichString)?;

terminal TEXT : '»' (!'«')* (EOF|'«');

It becomes quite straightforward once you have understood the escaping. Have a look at the last rule TEXT first: It says that a text starts with a closing French quote and ends with an opening quote or the end of the file. By inverting opening and closing quotes we mark up text instead of expressions.

A TemplateFile starts with a comment and switches to the expression mode for the preamble part consisting of the package declaration, the imports and the parameter declaration. The body is a RichString, which is an alternating sequence of RichStringLiterals and RichStringPart. The RichStringLiterals is essentially a text block (in inverted French quotes). The RichStringPart is either an Xbase expression, a RichStringForLoop or a RichStringIf. The latter inherit from the Xbase expressions with the same name to reuse as much of the Xbase infrastructure as possible. The rest should be easy.

Translation to Java

Each TemplateFile is compiled to a Java class with a generate method that takes a lambda expression as a parameter. The lambda expression is called to initialize the template's properties, by handling the template itself as an argument. This comes along nicely, especially when called from Xtend.

DSL

<!--«
  ...
»-->

Java

public class MyWebsite {
...  
  public String generate(final Procedure1<MyWebsite> init) {
    if (init != null)
      init.apply(this);
    String result = generate().toString();
    // remove leading -->
    result = result.replaceAll("^-->\\r?\\n","");
    // trim multi-newline to single newline
    result = result.replaceAll("\\r?\\n\\s*\\r\\n?", System.getProperty("line.separator"));
    return result;
    
  }
}

The corresponding code in the TemplateJvmModelInferrer (src) is:

class TemplateJvmModelInferrer extends AbstractModelInferrer {
...
     def dispatch void infer(TemplateFile element, 
                             IJvmDeclaredTypeAcceptor acceptor, 
                             boolean isPreIndexingPhase) {
       val simpleName = element.eResource.URI.trimFileExtension.lastSegment
       val qualifiedName = if(element.getPackage != null) 
           element.getPackage + "." + simpleName
         else 
           simpleName
    val javaClass = element.toClass(qualifiedName)
       acceptor.accept(javaClass)[
...
      // generate a method accepting an initializer lambda expression
      members += element.toMethod("generate", typeRef(String)) [
        parameters += element.toParameter(
          "init", typeRef(Procedures.Procedure1, typeRef(javaClass))
        )
        body = '''
          if (init != null)
            init.apply(this);
          String result = generate().toString();
          // remove leading -->
          result = result.replaceAll("^-->\\r?\\n","");
          // trim multi-newline to single newline
          result = result.replaceAll("\\r?\\n\\s*\\r?\\n",System.getProperty("line.separator"));
          return result;
        '''
...

Each Parameter becomes a Java property, i.e. a field with a getter and a setter.

DSL

param title = "No Title"

Java

private String title = "No Title";

public void setTitle(final String title) {
  this.title = title;
}

public String getTitle() {
  return this.title;
}

In the inferrer, note that we derive the property's type in three steps: Take the declared one, if there is none, derive it from the initializer and if even that fails, use String as default.

for (param : element.params) {
  val type = param.type 
    ?: param.defaultexp?.inferredType 
    ?: typeRef(String)
  members += param.toField(param.name, type) [
    if (param.defaultexp != null)
      initializer = param.defaultexp
  ]
  members += param.toSetter(param.name, type)
  members += param.toGetter(param.name, type)
}

The body of the template is compiled into a big private generate() method. We skip the inferrer code here, as it is straightforward. But we had to extend the compiler to support rich strings and the new FOR loop our new control structures. This is described in the next section.

DSL

»<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>«title»</title>
  <meta name="viewport" content="width=device-width, initial-sca...">
  <meta name="description"
    content="«description»">
  <meta name="author" content="«...

Java

private CharSequence generate() {
  StringBuilder _appendable = new StringBuilder();
  _appendable.append(ObjectExtensions.operator_elvis(
    "-->\n<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n\t<meta...",
    ""));
  _appendable.append(ObjectExtensions.operator_elvis(
    this.title,""));
  // appends galore
...

Extending the Compiler

We have added additional expressions to Xbase, so we have to tell the compiler how to translate them to Java. The XbaseCompiler (src) has a method doInternalToJavaStatement that dispatches the compilation with regard to the type of the current expression. This is where we have to hook in with our TemplateCompiler (src):

class TemplateCompiler extends XbaseCompiler {
  
  override protected doInternalToJavaStatement(XExpression expr, 
                                               ITreeAppendable it, 
                                               boolean isReferenced) {
    switch expr {
      RichString : {
...   }
      RichStringForLoop : {
...   }
      default :
        super.doInternalToJavaStatement(expr, it, isReferenced)
    }
  }
...

For a RichString, we declare a variable _appendable of type StringBuilder and append the results of all its evaluated expressions. Remember from the grammar that every second expression is a RichStringPart which can be null, so we use the 'elvis operator' ?: to insert an empty string in this case.

RichString : {
  val name = declareVariable(expr, '_appendable')
  newLine
  append('''
    StringBuilder «name» = new StringBuilder();
  ''')
  for (nestedExpression : expr.expressions) {
    nestedExpression.internalToJavaStatement(it, true)
    newLine
    append('''«name».append(ObjectExtensions.operator_elvis(''')
    nestedExpression.internalToJavaExpression(it)
    append(',""));')
  }
}

As our RichStringLiteral inherits from XStringLiteral (src), it does not need any special treatment. The same holds for RichStringIf and RichStringElseIif. The RichStringForLoop requires special treatment, because as opposed to the XForLoopExpression (src) that always returns null, we want it to return a concatenation of its results. This looks like

RichStringForLoop : {
  expr.forExpression.internalToJavaStatement(it, true)
  val paramType = typeProvider.getTypeForIdentifiable(expr.declaredParam)
  val name = declareVariable(expr, '_forLoopResult')
  newLine
  append('''
    StringBuilder «name» = new StringBuilder();
    for (final ''')
  serialize(paramType, expr, it);
  append(''' «declareVariable(expr.declaredParam, 
      makeJavaIdentifier(expr.declaredParam.name))» : ''')
  internalToJavaExpression(expr.forExpression, it)
  append(") {").increaseIndentation
    expr.eachExpression.internalToJavaStatement(it, true)
    newLine
    append('''«name».append(''')
    expr.eachExpression.internalToJavaExpression(it)
    append(');')
  decreaseIndentation.newLine.append("}")
}

The compiler now knows how to handle the new expressions in a statement context. In addition, we have to teach it to compile them in an expression context. This is what the second method does:

override protected internalToConvertedExpression(XExpression obj, 
                                                 ITreeAppendable it) {
  if (hasName(obj))
    append(getName(obj))
  else 
    super.internalToConvertedExpression(obj, it) 
}

As usual, we have to bind our TemplateCompiler (src) in the TemplateRuntimeModule (src) in order to be picked up as the XbaseCompiler (src) in the context of our language.

Type Computation

The type system has to know how to determine the types of our new expressions. This is the job of the TemplateTypeComputer (src): RichString becomes a StringBuilder. As opposed to its super type XForLoopExpression (src) a RichStringForLoop is of type StringBuilder as well. The for-loop's body is expected to have a type, as the results must be concatenatable, which is different from Xbase's for-loop.

class TemplateTypeComputer extends XbaseWithAnnotationsTypeComputer {
	
	def dispatch computeTypes(RichString expression, ITypeComputationState state) {
		super._computeTypes(expression as XBlockExpression, state)
		state.acceptActualType(getTypeForName(StringBuilder, state))
	}
	 
	def dispatch computeTypes(RichStringForLoop expression, ITypeComputationState state) {
		super._computeTypes(expression as XForLoopExpression, state)
		state.acceptActualType(getTypeForName(StringBuilder, state))
	}
}

Like the compiler, we have to bind this implementation in our runtime module (src) as well.

Value Converter

The RichStringLiterals still have the French quotes around their values. As we do not want to see them in the output, we have implemented the TemplateValueConverterService (src) and bound it in the runtime module (src).

Content Assist

The French quotes are not easy to type on every keyboard. We have adapted content assist to insert them when the cursor is inside a TEXT terminal:

public class TemplateProposalProvider 
    extends AbstractTemplateProposalProvider {
  @Override
  public void complete_TEXT(EObject model, 
                            RuleCall ruleCall, 
                            ContentAssistContext context,
      ICompletionProposalAcceptor acceptor) {
    acceptor.accept(new CompletionProposal("«»", 
        context.getOffset(), 0, 1));
  }
}

Syntax Highlighting

Sometimes it is hard to see whether you are in text mode or in expression mode. To give the user better feedback, we have changed the way the text is highlighted. This customization consists of two parts: Add new highlighting styles in the TemplateHighlightingConfiguration (src) and apply them to the text in the TemplateHighlightingCalculator (src). As this is rather extensively covered in the Xtext documentation, we skip a deeper explanation here.

Little Tortoise


Do you remember the programming language Logo? Logo was used in computer science classes to teach children how to program. In fact, it was a adaptation of LISP! But the remarkable part was the so-called turtle, a graphical cursor that can be given commands to move and turn, thereby drawing lines.

The goal is a language to control a turtle drawing an image. Technically, this example will teach you how to adapt and use the XbaseInterpreter (src) for your own languages.

Overview

We have built a language that allows to define Programs and SubPrograms. Each of these has a body, which can contain any number of expressions. In addition to the standard Xbase expressions, we are able to issue commands to the tortoise. Here is an example explaining the concepts in comments:

// Program: Haus vom Nikolaus
begin
  val length = 150                // local variable
  val diagonal = length * sqrt(2) // all Math.* methods are available
  lineWidth = 2                   // assignment of a property
  square(length)                  // call to a SubProgram
  turnRight(45)                   // call to a command method
  lineColor = blue                // all ColorConstants.* are available
  forward(diagonal)
  turnLeft(90)
  lineColor = red
  forward(diagonal / 2)
  turnLeft(90)
  forward(diagonal / 2)
  turnLeft(90)
  lineColor = blue
  forward(diagonal)
end  // main program

sub square           // a subprogram
  int length         // parameter
begin
  for (i : 1..4) {   // loop-expression from Xbase 
    forward(length)
    turnRight(90) 
  }
end  // sub square

The main trick about our solution is to not bake in the turtle commands into the language itself, but define it in the runtime library. This way, the language stays as slim as can be and additions can be easily added without the need to regenerate the whole language infrastructure.

The core of the runtime library is the class Tortoise (src). You can think of it as of our only domainmodel class: It keeps the current state of the tortoise and allows modifying it using methods. Here is an excerpt of its code:

class Tortoise {
  double angle
  double x
  double y
  @Accessors int delay = 200

  boolean isPaint = true
  @Accessors int lineWidth
  @Accessors Color lineColor

  List<ITortoiseEvent.Listener> listeners = newArrayList()
...

When a method changes the state of the tortoise, an event is thrown. These events are consumed by a GEF based view and turned into animations of a TortoiseFigure (src). This loose coupling of model and view allows for easier testing.

Running the Example

In the runtime Eclipse, open the Tortoise View (Window > Show View > Other > Xtext > TortoiseView). Then open one of the example files in org.eclipse.xtext.tortoiseshell.examples. The Program is interpreted on editor activation and on save. An additional toggle button Step Mode in the Tortoise View allows to execute the code live from the editor up to the caret's current line.

Tortoise takes a rest after running the Pythagoras example

Grammar

The grammar is very short. Once again, we inherit from the Xbase language to have nice Java integration and rich expressions. A user can define a Program, which can have SubPrograms with parameters. The Executable rule is never called, but defines a common supertype for Program and SubProgram that will hold their common member body. A Body is an XBlockExpression from Xbase, but with the keywords begin and end instead of the curly braces.

grammar org.xtext.tortoiseshell.TortoiseShell
  with org.eclipse.xtext.xbase.Xbase

import "http://www.eclipse.org/xtext/xbase/Xbase"
generate tortoiseShell "http://www.xtext.org/tortoiseshell/TortoiseShell"

Program :
  body=Body
  subPrograms+=SubProgram*;
  
SubProgram:
  'sub' name=ValidID (':' returnType=JvmTypeReference)?
  (parameters += FullJvmFormalParameter)*
  body=Body;

Body returns XBlockExpression:
  {XBlockExpression}
  'begin'
  (expressions+=XExpressionInsideBlock ';'?)*
  'end';
  
Executable:
  Program | SubProgram;

Translation to Java

With the tortoise commands defined as methods in the runtime library class Tortoise (src), we have to infer a Java class that inherits from this. Within this class, we create a method for each Program and SubProgram. The resulting code looks like this:

class TortoiseShellJvmModelInferrer extends AbstractModelInferrer {
  public static val INFERRED_CLASS_NAME = 'MyTortoiseProgram'
  
  @Inject extension JvmTypesBuilder
  
  def dispatch void infer(Program program, 
                          IJvmDeclaredTypeAcceptor acceptor, 
                          boolean isPreIndexingPhase) {
    acceptor.accept(program.toClass(INFERRED_CLASS_NAME))[
      superTypes += typeRef(Tortoise)
      if(program.body != null)
        members += program.toMethod("main", typeRef(Void.TYPE)) [
          body = program.body
        ]
      for(subProgram: program.subPrograms)
        members += subProgram.toMethod(subProgram.name, 
            subProgram.returnType ?: inferredType(subProgram.body)) [
          for(subParameter: subProgram.parameters)
              parameters += subParameter.toParameter(subParameter.name, subParameter.parameterType)
          body = subProgram.body
        ]
    ]
  }
 }

Interpreter

The Xbase language library does not only provide a compiler that generates Java code, but also an interpreter (src). This has been adapted to execute our Programs.

After all an interpreter is just a big visitor. For each expression type, it has an evaluation method, that recursively calls the evaluation methods for the subexpressions for its arguments. The methods also pass an execution context storing all temporary state such as local variables.

The first thing we have to cope with is the mixture of existing Java methods (from the super class Tortoise (src)) and inferred ones. While the former are evaluated via Java reflection, we need special treatment for the latter. The idea is to bind an instance of Tortoise (src) to this and intercept calls to the inferred methods to execute them directly. This is accomplished by overriding the method invokeOperation:

@Inject extension IJvmModelAssociations
 
override protected invokeOperation(JvmOperation operation, 
                                   Object receiver, 
                                   List<Object> argumentValues) {
  val executable = operation.sourceElements.head
  if (executable instanceof Executable) {
    val context = createContext
    context.newValue(QualifiedName.create("this"), tortoise)
    operation.parameters.forEach

p, i| context.newValue(QualifiedName.create(p.name), argumentValues.get(i))

@Xtext on Twitter