Finally, I have finished editing the whitepaper on various debugging techniques in JetBrains MPS. Thanks go out to Sergej Koščejev for contributing to this whitepaper. You should also check out his complementary post at specificlanguages.com.
Like most programs, MPS writes messages to a log file — lots of messages. To make sense of so many messages, it helps to use a tool that improves the visual presentation, for example:
- grid display: separate fields and align them nicely, like timestamp, logger, message
- color coding for message severity, like bright red for ERROR
- set column order: move a column out of view if it is not relevant to you.
- filter out messages, like: block all messages from logger com.microsoft.alm.
Apache Chainsaw is such a tool.
Why choose Apache Chainsaw? First of all it is free. Second, it is a companion to the log4j library that is used by MPS. Chainsaw is designed to play nice with log4j. Third, it has been around for a long time, so it can be expected to offer relevant functionality. As a disclaimer: I myself have not used chainsaw before.
How to configure Chainsaw? Let’s make a choice here: we are going to connect Chainsaw to the MPS log via the file system, not via sockets. Sure, the log4j architecture offers a socket based approach, but that is much more involved. It would require configuration changes at the MPS side of things, whereas using the log file only requires a proper configuration at the Chainsaw side. Moreover, changing the configuration at the MPS side has the difficulty that the log4j version that MPS uses is way beyond end-of-life, making it difficult to find the proper components and documentation. MPS still uses log4j version 1.2.17 released Jan 2013 and log4j version 1 was declared end of life in 2015. I briefly tried to get the sockets running anyway but I did not succeed. Maybe at a later time.
It proved not to be very difficult to get Chainsaw to connect to the MPS log file. First, a short explanation if you want quick results. Second, a longer explanation if you want to know how this works under the hood. Finally the details if you want to explore further.
HOWTO – the short explanation
This XML snippet tells Chainsaw where to find the MPS log file and how to parse it.
<plugin class="org.apache.log4j.chainsaw.vfs.VFSLogFilePatternReceiver" name="MPS"> <param name="appendNonMatches" value="true" /> <param name="autoReconnect" value="true" /> <param name="fileURL" value="file:/C:/Users/jgee/.MPS-126.96.36.199-c743b9d/system/log/idea.log" /> <param name="logFormat" value="TIMESTAMP [PROP(RELATIVETIME)] LEVEL - LOGGER - MESSAGE" /> <param name="name" value="MPS" /> <param name="promptForUserInfo" value="false" /> <param name="tailing" value="true" /> <param name="timestampFormat" value="yyyy-MM-dd HH:mm:ss,SSS" /> <param name="waitMillis" value="2000" /> </plugin>
- Adapt the ‘fileURL’ parameter to your situation: 1/ change the user name “jennek” to yours 2/ change the MPS version “2019.1” to yours. Check if you can find the log file at this path. If it is not there, please read the longer explanation
- Open the default configuration file of Chainsaw and add the XML snippet. This file is named “receiver-config.xml” and it should be in folder “.chainsaw ” in your home directory. If it is not there, please read the longer explanation.
- Restart Chainsaw. You should now see a tab named
file-/C:/Users/jennek/.MPS2019.1/system/log/idea.log. Tweak the view to your liking using the menu item “Tab Preferences” under the “Current Tab” menu. Your tweaks are persisted by Chainsaw.
One more thing. In the logger column, you may see weird entries like
m.microsoft.alm.plugin.operations.BuildStatusLookupOperation. This is caused by MPS using a 60 character limitation on the logger name, truncating
m.microsoft. This results in quite a few spurious loggers in the Logger Tree Pane on the left. If you want to fix this, do this:
- Open <MPS-home>/bin/log.xml in your favorite editor
- Find the appender element with
<param name="file" value="$LOG_DIR$/idea.log"/>
- Just below the line find this line
<param name="ConversionPattern" value="%d [%7r] %6p - %30.30c - %m \n"/>
- Remove the
30.30which means: “minimum field width 30, maximum field width 30 more”, and be sure to keep the
Now MPS will write the logger name to the log file without truncation.
Under the Hood – the longer explanation
To understand how Chainsaw should read the log file, let us look at how MPS writes the log file. As said, MPS uses log4j version 1. The log4j configuration is typically found in a file named log4j.xml but in the MPS distribution you will not find this file. Instead MPS has a file named log.xml in the /bin directory that is very similar to a log4j.xml file. This file contains a number of path variables that need to be expanded. Only after expansion the log.xml file is passed to log4j. To see this in action, read the gory details down below.
The file log.xml contains this piece of xml
<appender name="FILE" class="org.apache.log4j.RollingFileAppender"> <param name="MaxFileSize" value="1Mb" /> <param name="MaxBackupIndex" value="3" /> <param name="file" value="$LOG_DIR$/idea.log" /> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%d [%7r] %6p - %30.30c - %m \n" /> </layout> </appender>
This is the part that tells log4j to append log messages using a rolling file mechanism. The file parameter with value $LOG_DIR$/idea.log says to which file to append– where the $LOG_DIR$ is expanded. The ConversionPattern tells that the log message must be formatted in a specific way. This is precisely the information that Chainsaw needs to find the file and to split the log line into separate fields.
Here Chainsaw makes life easy for us. It would be a burden to first look at the conversion pattern of the appender that combines the message fields into a line and then to construct the inverse pattern to unravel this line into separate fields. Chainsaw will do this for us. Try this in Chainsaw (see screenshot below):
- Pick menu item “Load Chainsaw Configuration” under menu “File”
- Select “Use fileappender entries from a log4j config file”
- Press”Open File…” and navigate to the log.xml file in the bin folder of your MPS install
- Press “Save configuration as…” and navigate to some temporary folder and save as -say- mps.xml
Open mps. xml in your favorite editor and there you have it: Chainsaw has written the receivers (plugin) for the senders (appenders) in log.xml. The plugin we need is the one with “$LOG_DIR$/idea.log” in the element with name “fileURL”. See below.
<plugin class="org.apache.log4j.chainsaw.vfs.VFSLogFilePatternReceiver" name="FILE"> <param name="appendNonMatches" value="true" /> <param name="autoReconnect" value="true" /> <param name="fileURL" value="file:/C:/path/to/chainsaw/bin/$LOG_DIR$/idea.log" /> <param name="logFormat" value="TIMESTAMP [PROP(RELATIVETIME)] LEVEL - LOGGER - MESSAGE " /> <param name="name" value="FILE" /> <param name="promptForUserInfo" value="false" /> <param name="tailing" value="true" /> <param name="timestampFormat" value="yyyy-MM-dd HH:mm:ss,SSS" /> <param name="waitMillis" value="2000" /> </plugin>
But before we can use it we need to fix a few things
- in the element with the name “fileURL”, replace the path with $LOG_DIR$ with the path on your system. See the HOWTO section above.
- in the element with name “logFormat”, delete the ” ” — this is a bug. If you keep it, the splitting of a log line into fields fails and the entire log line ends up in the message column in Chainsaw
After these edits you should have an XML snippet at the beginning of this post. This snippet you may include in a Chainsaw configuration file — either the one that is loaded when Chainsaw starts up or one that you load on demand. To find the start-up configuration of Chainsaw, do this:
- Pick menu item “Show Application-wide Preferences…” under menu “View”
- use the path in the “Auto Config URL” under the tab “General”, typically
Logging log4j itself – for further exploration
If you want to explore how the log4j system is behaving, try this:
- find the file that launches MPS, on Windows this is <MPS-home>/bin/mps.bat
- where you see javaw.exe, remove the w and keep the java.exe.
This keeps the console window open while running MPS, showing all the logging from MPS. However this may slow down the IDE considerably. This is one good reason to view the log with Chainsaw instead. Now when you start up MPS you see that the very first lines are:
log4j:WARN No appenders could be found for logger (jetbrains.mps.util.ClassPathReader).
log4j:WARN Please initialize the log4j system properly.
for more info.
Apparently log4j is looking for the default log4j.xml configuration file and complains it cannot find it. This is by design as discussed above.
Next try this:
- edit <MPS-home>/bin/mps.bat again
start "" "%JAVA_EXE%" %ALL_JVM_ARGS% -Dlog4j.debug -Didea.main.class.name=...
This forces log4j to provide logging at the debug level. Now when you start up MPS you see lines like
log4j: Class name: [org.apache.log4j.RollingFileAppender]
log4j: Setting property [maxFileSize] to [1Mb].
log4j: Setting property [maxBackupIndex] to .
log4j: Setting property [file] to [C:\Users\jennek\.MPS2019.1\system\log/idea.log].
log4j: Parsing layout of class: "org.apache.log4j.PatternLayout"
log4j: Setting property [conversionPattern] to [%d [%7r] %6p - %30.60c - %m
log4j: setFile called: C:\Users\jennek\.MPS2019.1\system\log/idea.log, true
log4j: setFile ended
This indicates that log4j actually now is processing the log.xml file. This info is useful when you start tweaking the MPS log.xml file.
Finally, in case you wondered, the log window in the IDE is also part of the log4j configuration but there is no entry for it in the log.xml file. The appender for this is created programmatically while the IDE is launching ( source: https://mps-support.jetbrains.com/hc/en-us/community/posts/207463965-Logging-Mechanism-in-MPS)
The com.dslfoundry.plaintextgen extension (https://jetbrains.github.io/MPS-extensions/extensions/plaintext-gen/) for JetBrains MPS (https://www.jetbrains.com/mps/) enables a language designer to write model-to-text generation templates in pretty much the same way one writes model-to-model transformations in MPS. This is very useful for many reasons, the most prominent two of which are in my opinion: you don’t have or don’t want to build a generation target language for your text generation needs (e.g. because you need to make some model-to-text transformation very quickly) or if you want to work more like classical template-based text generation similar to Xtext.
This tutorial describes some of the most common uses of the plaintextgen.
This tutorial assumes that you have basic knowledge of MPS, i.e. how to create a new project and how to make a language with an editor and a generator.
How to use plaintextgen?
Before you can use plaintextgen, you need to install it first. You can either do so by downloading the MPS Extensions (https://github.com/JetBrains/MPS-extensions/releases) and copying them into your <MPS_Installation_Folder>\plugins folder, adding them to your global libraries (File –> Settings –> Build, Execution, Deployment –> Global Libraries), or by installing the com.dslfoundry.plaintextgen plugin from the MPS Marketplace (File –> Settings –> Plugins –> Marketplace).
Once you have com.dslfoundry.plaintextgen installed, you can use it within a generator template. The idea behind plaintextgen is to reproduce the look-and-feel of the textual editor aspect as much as possible, since both text generator and textual editor in principle handle the rendering of concrete syntax for a textual notation on your model. Therefore, I am not going to explain each detail and you can try to use the basis of your mental model of textual editors (e.g. vertical and horizontal collections, newlines, surrounding blocks with collections) on the plaintextgen language. Names are a bit different, e.g. vertical collection (from editor) versus VerticalLines, horizontal collection versus Line, CellModel versus Word, etc. You can see the metamodel of plaintextgen in the structure of the language.
If you feel that certain things are very different from the editors aspect, then this is most probably a bug or not-yet-implemented feature. Feel free to file an issue (or even better: make a pull request) on https://github.com/JetBrains/MPS-extensions for the plaintextgen component.
I will illustrate most of plaintextgen’s basic functionality by extending the generator of the example language com.dslfoundry.plaintextgen.example.testlang. What I will do is add a generator for a C++ file, based on an existing C++ listing from the ThinkC++ book by Allen B. Downey ( http://greenteapress.com/thinkcpp/index.html ). The listing in question is one of the accompanying code samples on github: https://github.com/AllenDowney/ThinkCPP/blob/master/code/cha11/cha11.cpp . The templating of the code is only to illustrate the mechanism and will probably be completely nonsensical, so don’t try to think too hard on what the end result would mean or whether it can be compiled 🙂 Now, if you have any suggestions for improvements, I’ll be happy to hear that in the comments.
If you want to reproduce this tutorial based on com.dslfoundry.plaintextgen.example.testlang, you can clone the https://github.com/JetBrains/MPS-extensions repository and open the MPS project under the code directory. The MPS version that is supported on master can be found at the key ext.mpsMajor in the file build.gradle in the root of the repository. If you need the code for an older version, there are maintenance/mps<version> branches. States of the branches are described at https://jetbrains.github.io/MPS-extensions/#current-versions. One last thing to add before we begin: I will add MPS links that you can click from here if you have the code project of the MPS Extensions repository open and it will open the specific node that the link points to.
So let’s get going!
Just for reference, com.dslfoundry.plaintextgen.example.testlang can be found here in the project:
First we will have a brief look at the test model (instance of the TestConcept concept) in com.dslfoundry.plaintextgen.testlang.sandbox:
So we have a TestConcept instance which has a property called testProperty with value TestValue. Then, it also has two children in its list of testSubConcepts with (respectively) testProperty values TestValue2 and TestValue3.
Let’s make our first change to the language by adding a new plaintextgen-based root template to its generator:
So far nothing special, it’s just the creation of a root template. The only thing you need to know is to import the com.dslfoundry.plaintextgen language (Ctrl+L+L and search for the language, then select it and press enter or double click it).
Now, we will import the text from https://github.com/AllenDowney/ThinkCPP/blob/master/code/cha11/cha11.cpp into our newly created root template:
So we only need to select the plain text from the C++ source file and then do execute the Paste Unstructured Text on the plaintextgen root template. Plaintextgen takes care of analyzing indentations and makes a full plaintextgen template for you.
Next, let’s feed the testProperty of the root TestConcept into the template:
This works again pretty much the same as the normal MPS model-to-model generator templating mechanism. We just add a property macro on the filename based on the testProperty value of the root TestConcept and then we add a filename extension .cpp. Then we run a build and perform a Preview Generated Text action (via right-click) on the model root. The result is a text file called TestValue.cpp which lists the hard-coded contents that we have imported in the last step.
To continue, we will feed the value of testProperty of the root TestConcept to the struct name:
Here we use the Split-extract Selection into Word intention of plaintextgen in order to make it a separate word that can be templated. Then we put a node macro for testProperty on the split-extracted word. When we build this and then again run a Preview Generated Text action, we see that the name Time has been replaced by TestValue, as would be expected.
As a next step, we will feed the testProperty values of the testSubConcepts children ( TestValue2 and TestValue3 in the model) to the struct members:
To do this, first we delete all but one of the struct members from the template. Now, we apply a LOOP macro on the template (again, just normal MPS generator macro stuff); N.B.: don’t forget to include the newline, or else you will miss newlines in the output. We finish this step off by doing a split-extract on the name of the member and feeding the testProperty value to it via a property macro. When we build this, the output is as expected: we have two members called TestValue2 and TestValue3.
Finally, we are going to add an implementation for each member based on the testProperty names of the testSubConcepts children:
Again, we delete all implementation functions except for one. Since we cannot apply a macro on multiple nodes, we need to wrap the multiple lines of the implementation function into one node. To do this, we use the Surround with Vertical Collection selection intention (N.B.: we did not press the light bulb, but pressed Alt+Enter to trigger the intention since it’s a special intention which can be executed on a selection of multiple nodes, provided by the language de.itemis.mps.selection.intentions). Now that the multiple lines are wrapped, we can apply a LOOP macro, do again the split-extract on the name of the implementation, and use a property macro to feed the value of the testProperty of each of the testSubConcepts children.
Of course, we could continue and really make this a much more refined example, but this should be enough for now to illustrate the major mode of working with plaintextgen to do your own templates.
A last remark: this tutorial took a template imported from plain text (a C++ file) as a basis and did some plaintextgen operations on it to get to the desired results. Of course, you can also build your own plaintextgen templates from scratch (adding new word, vertical lines, lines, etc.), but if you have a textual source, then I think the above method is much easier.
Often times, when creating a new language, I skip the “Create Sandbox Solution” checkbox. Recently, I have learned that the sandboxes are actually not just solutions with the name sandbox. Because of their property of actually residing in the language folder itself, many MPS Extensions are actually using sandboxes to package example models with a language to show its use.
Now imagine that you have created a language without a sandbox, then you may wonder how to add one. Although the answer is very simple, I still describe it here, since it may not be immediately obvious.
I’ll following the com.jetbrains.etc naming convention in this example. Imagine you have a language called com.dslfoundry.mylanguage and which doesn’t have a sandbox yet. To add a sandbox to it:
- Create a new solution called com.dslfoundry.mylanguage.sandbox
- Add a new model to it called com.dslfoundry.mylanguage.sandbox (so normally, you just do an “add model” and then remove the dot at the end of com.dslfoundry.mylanguage.sandbox.)
- Open a file explorer and navigate to the folder of your language (usually that is <project_folder>/languages/com.dslfoundry.mylanguage)
- Move the entire folder of the earlier created com.dslfoundry.sandbox solution to your language folder
- Now rename com.dslfoundry.sandbox to just sandbox
- To make the sandbox show up on your project, you have to right-click your project in the logical view and select properties, then click the + to add the .msd-file of the sandbox solution.
MPS comes with pretty good default settings, but there are some settings that I really like to change as soon as I have a fresh MPS IDE in front of my nose. Below is a description of such settings.
Hide empty aspect tabs
By default, MPS always shows all tabs for all aspects (also aspects that are empty) for each concept. I have found that I keep more overview when the empty aspects of a concept are hidden. You can change this, by setting the Aspect Tabs option to Each aspect node in a separate tab:
Hide generator warnings that you cannot fix
So far (right up until MPS 2019.1), MPS shows some really annoying “Failed to replace dynamic reference” generator warnings. They are annoying because they pollute the build logging and you cannot do anything about them as a language engineer. You can remove these (without being afraid that your languages will not be checked when you commit, because MPS checks all modified languages and models on every commit) by unchecking Show warnings in the generator settings:
Disable language checking in every build
When working incrementally on a language implementation, I often run many rebuilds with every small delta that I want to test for my language. Since model/language checking is not incremental, having a check every time when I need to build a language can take a lot of cumulatively collected overhead time. Therefore, I like to, by default, turn off the model checking for every build.
There is a caveat, though: since MPS’ language development workflow currently relies on languages/models being checked in the IDE and not in command line/buildserver build, you should do model checking before checking in! I still change the default setting, which requires more discipline from me, but saves me some time.
Disabling continous modelchecking can be accomplished by unchecking the item Build –> Check models before generation item in the menu:
Prevent auto-opening of last-opened project
For me, the reason of shutting down or restarting MPS is usually because I need to switch between projects or because I need to play with certain settings. This makes the default feature of reloading your last opened project upon MPS startup not very useful. I turn it off here:
This piece of information is provided thanks to Hristina Moneva (see coments).
The memory indicator does not only give you a way to see how much memory is allocated to the VM that MPS and your memory use, but it also gives you a way to do garbage collection manually. It is located in the lower right corner and looks like this:
Adding the memory indicator can also be done via the settings menu:
Today, I released a first version of my slides explaining JetBrains MPS generators, including useful generator patterns: https://github.com/eugenschindler/mps-teaching/blob/master/JetBrains-MPS-generators.pdf
Use it to your advantage!
Today, I am going to lead you through a tutorial of how to build a graphical (slightly UML-like) editor for the structure of an MPS language. The main goal of this tutorial is to show you how to use the diagram editor and querylist plugins that have been developed by Itemis as part of the mbeddr platform, without which building such a graphical structure editor would be at the least very much effort. It is assumed that the reader knows at least MPS basics (languages, structure, editors, etc.).
The code for this tutorial can be found at https://github.com/DSLFoundry/mps-graphicalstructureeditor and the MPS used contains at least the mbeddr.platform plugins (which contain querylist and diagram editor). An earlier post deals with the installation of these plugins into MPS.
As a side note: Itemis and JetBrains (and more contributors, such as DSLFoundry) are currently collecting and maintaining very useful plugins, such as the diagram editor in the MPS Extensions (more documentation here).
So, before we start with the nitty gritty details, let me explain to you what I mean by a graphical structure editor: let’s recall that the MPS structure aspect is at the core of language definition. With the current structure definition, each concept has a structure definition. An example of the java if-statement concept structure definition from MPS baseLanguage can be seen here:
Let’s create a simple language (let’s say we call it com.dslfoundry.testlanguage, or testlanguage for short) that we will experiment with while building our graphical structure editor. The following diagram shows the structure of testlanguage:
We have various concepts (TestConcept1, TestConcept2, and TestConcept3), interface concepts (TestIC, TestIC2, TestIC3) as well as relationships, such as implementation (e.g. TestConcept1 implements TestIC2 and TestConcept3 implements TestIC3), extension (e.g. TestIC2 and TestIC3 extend TestIC), containment (e.g. TestConcept1 has a list of children called tc2s which are of conceptTestConcept3), and reference (e.g. TestConcept1 has a reference called tc3 which is of concept TestConcept2).
Normally, one would have to define the structure of TestLanguage by filling in the structure aspect for each concept. For example, for TestConcept1, the structure would be:
But we are not going to use the “normal” way to define the structure. Instead, we will make a new language called com.dslfoundry.graphicalstructureeditor (or graphicalstructureeditor for short), with which we will be able to (with some limitations) edit the structure of testlanguage.
Let’s first make some decisions on how we want the graphical structure editor to look like and how we want it to be used. For the looks, let’s go for a box-line “roughly UML-like” look, i.e. concepts and concept interfaces are represented as boxes and relations between concepts are represented as labeled lines. Now, for the usage: since we want to edit the actual structure of a language, it makes sense to extend the language in MPS that is used to define the structure aspect. This will give us a simple entrypoint for our new language, enabling us to just right-click on a language’s structure and add an editable diagram to the structure Let’s first find out which language we want to extend: if we go to a concept (e.g. TestConcept1 of testlanguage) and then visit the inspector, we can see that a concept’s declaration is an instance of jetbrains.mps.lang.structure.structure.ConceptDeclaration:
So the language we want to extend with the graphicalstructureeditor language is jetbrains.mps.lang.structure. We can just add this as an extends dependency to the graphicalstructureeditor dependencies tab under language properties (select graphicalstructureeditor language and Alt+Enter, or right click –> properties).
We’re going to choose to have one concept representing the language structure diagram. Let’s call it ConceptStructureRelations (right-click the structure of the graphicalstructureeditor language and choose new concept):
Now, since the diagram editor needs to render concepts, we will start defining relations, which leads to the rest of the graphicalstructureeditor language design:
Just to highlight what the above language design is about: the ConceptStructureRelations diagram contains relations (of concept AbstractConceptRelation) and interface relations (of concept InterfaceConceptExtendsRelation). There are a number of concrete relationships extended by AbstractConceptRelation, namely: AbstractConceptChildRelation, AbstractConceptReferenceRelation, ConceptExtendsRelation, and ConceptImplementsRelation. The finished structure declaration is left as an exercise for the reader or can be found in the example code.
Now that we have defined the structure of our graphicalstructureeditor language, we have the context to start using diagram editor, which is one of the topics of this tutorial. But before we can do that, we need to set up a little bit more context: since we want to represent a ConceptDeclaration in an alternative notation (namely as a box in a diagram) without interfering with the existing “normal” textual notation of ConceptDeclaration, we have to add some editor hints to our language, so that we will be able to switch between the graphical and the textual notation. We do this by adding a hints definition called graphicalStructureHints (right-click the editor aspect of the graphicalstructureeditor language and choose New –> ConceptEditorContextHints):
Now, let’s start by making the editor for ConceptStructureRelations, which is supposed to be a diagram editor. When making an editor for a concept (e.g. double click the concept in the logical view, go to the editor aspect tab and choose new concept editor), the initial editor definition (after clicking on the <default> text in front of the “editor for concept” text and choosing the graphical hint: Ctrl/Cmd+Space –> graphical) looks like this:
We can start the normal way by adding the title for the diagram, e.g. by typing “Concept Structure Relation Diagram” which will turn the text into a constant in the editor and then surrounding it by an indent collection (Alt+Enter –> Surround with Indent Collection). Now we add another cell (press Enter) and this is where we want to insert a diagram. To do this, we first import the diagram editor language (Ctrl+L+L, search for de.itemis.mps.editor.diagram and select it for import). Now we can choose diagram, which will insert a diagram cell:
Let’s put the diagram on the next line (go to the constant text cell before the diagram, press Alt+Enter –> Add new line) and we get:
Let’s start with specifying what the contents of the diagram will be. We will want to show all concepts (ConceptDeclaration from jetbrains.mps.lang.structure), all interfaces (InterfaceConceptDeclaration from jetbrains.mps.lang.structure) as well as all relations (which are the lists of relations and ifrelations) in the graphicalstructureeditor language. The latter two can simply be completed from the rolenames of the ConceptStructureRelations concept (for which we are currently defining an editor). Since we don’t have the full set of concepts and interface concepts accessible from this concept, we need to somehow query them together. The diagram editor language provides a nodesQuery construction for that where we can just plugin an smodel query. But what smodel query to write? Let’s start with a nodesQuery for all ConceptDeclarations: imagine that we have a ConceptStructureRelations diagram as part of a language’s structure. Then we would want all the ConceptDeclarations of the language’s structure. Now we need a little bit of MPS architecture knowledge: MPS doesn’t really make a difference between languages and solutions; both are actually modules which contain models. So a language is just a module and each of its aspects is a model. That means that the structure of a language is a model. This knowledge enables us to write an smodel query, which will simply be node.model.roots(ConceptDeclaration), meaning “all the nodes that are instances of ConceptDeclaration in the model where node (in this case the diagram) resides”. The InterfaceConceptDeclarations are queried in an analogous way, leading to the following total specification for the diagram contents:
Before we go further with the diagram editor, let’s first add a box editor to the ConceptDeclaration concept. Since we are going to add an editor in our language for a concept that is not in our language, but in jetbrains.mps.lang.structure, we just right-click the editor aspect of the graphicalstructureeditor language and choose New –> Concept Editor. Then we choose again the graphical hint (see earlier) and ConceptDeclaration after the text “editor for concept”. Now, similar as for the ConceptStructureRelations, we add a cell from the diagram editor, but instead of the diagram cell, we choose the diagram.box cell:
The resulting editor will look as follows:
For the editor of the box, we can get inspired by the original ConceptDeclaration editor (e.g. find it by pressing Ctrl/Cmd+N+N, searching for ConceptDeclaration_Editor and pressing Enter): just copy the editor from there and paste it into the editor part of the diagram.box. Then you can remove or add whatever you like. A final version of it could look something like this:
Since we are making new editors anyway, we can also add en editor for a relation (we will only do one, since the other relations are pretty much analogous). Let’s pick AbstractConceptChildRelation: it has a bunch of references (from AbstractConceptDeclaration (which could be ConceptDeclaration or InterfaceConceptDeclaration) inherited from AbstractConceptRelation, to AbstractConceptDeclaration, and linkD which is a LinkDeclaration – a child of AbstractConceptDeclaration, which could represent either a child or a reference in a language’s concept or concept interface). Again, create an editor for AbstractConceptChildRelation and add in a diagram.edge:
The resulting editor will look like this:
At the from section, we can fill in the target, which is the from of AbstractConceptChildRelation. The diagram editor language has a special keyword for this, called thisNode. So the target of from is a box that points to thisNode.from (so the box at the side of the from reference). We also want to fill in role cell, since that allows us to show multiplicities on both sides of the child relationship. For the from side, this is easy: since a parent always can contain zero or more children, the cardinality at the parent-side is 1. At the to-side, this is more complicated, however: the structure definition has the information about the multiplicity at the child side (it is stored in node.from.linkDeclaration.sourceCardinality).
Now we have a problem: with the normal MPS editor mechanism, you cannot project properties or editors that are not a direct child or reference of the concept’s editor. Sure, you can use a read-only model access, but that access will be read-only. Aha, but you could use a read-write model access, right? Sure, but with it you can only use strings and it is quite less safe than just plugging in an editor construct like a property, reference or child projection. To solve this problem, enter the com.mbeddr.mpsutil.editor.querylist extension from Itemis. Querylist allows you to just project properties, children or references from any node that you can find from the current node by means of an smodel query. In order to start using querylist, press Ctrl/Cmd+L+L, search for com.mbeddr.mpsutil.editor.querylist and then press Enter to import it. Now you can add a querylist at the role cell of the to section (use only querylist; all the other query* elements are deprecated, since you can to everything with the querylist construct):
To query for the concept whose editor you want to access, you have to click on querylist and open the inspector (e.g. Alt+2):
So the smodel query specifies that we want to be able to project properties, children or references from the LinkDeclaration that is connected to the from reference of the current node (which is an AbstractConceptChildRelation). The node.linkD.index is a way to find the linkDeclaration that is coupled to the relevant line in the diagram.
Now we can choose to project sourceCardinality (Ctrl+space in the querylist cell model and choose sourceCardinality from the completion menu). The final version looks like this:
OK, these were the hardest parts of building a graphical structure editor for MPS. Of course there is some more operational code to add to really make it work, so I will give some pointers on how to that. But first, let’s see how an editor for TestLang would look like when it’s finished:
In order to be able to drag & drop concepts and interface concepts onto the diagram canvas, you need to specify palette entries in the ConceptStructureRelations_graphical_Editor. See the code sample on how to do this. Once you have done this, you get entries like this in the diagram palette:
For each of the relations, you need a way to create the relation. The diagram editor has anchors on the boxes in order to drag a relation from box to box, e.g.:
The picture above shows that if you point on the +, you can create 4 types of relationships (C for child as a target, R for reference as a target, E for extend, and I for Implement). You need to add code for that. In the sample, you can find this code under the connection creators section of ConceptStructureRelations_graphical_Editor.
You need to specify what happens if you delete a relationship. For an example of such code, see the delete section of the AbstractConceptChildRelation_Editor.
You need to add editor for the other types of relations that were described in the graphicalstructureeditor language design. Finished versions of these can be found in the code sample.
When you create an instance of a ConceptStructureRelations for the first time in a language of your choosing (see an example in testlanguage), then it will look very disappointing, like this:
What you need to do in this case, is to push the right editor hint:
This will show the diagram editor.
Bonus: you can write an intention on the ConceptStructureRelations concept that reads the structure of a language and synchronizes the diagram with it (useful for adding a diagram to an existing language or to update the diagram when you’ve edited in the “normal” way). An implementation of this can also be found in the code sample.
What’s left to do on an actual graphicalstructureeditor implementation: the graphical structure editor language that is described in this tutorial exists at https://github.com/DSLFoundry/mps-graphicalstructureeditor. Although it already makes it possible to do some basic language structure editing, it also has some limitations. For example, there is currently no way to edit extensions to languages outside the “current” language (i.e. the language in which the ConceptStructureDiagram is situated) graphically, since this involves making design choices on how to represent this in the diagram (such as: how much of the context outside the current language must be shown?). There is curently a list of issues for the graphical structure editor which can be found here: https://github.com/DSLFoundry/mps-graphicalstructureeditor/issues. If you feel like contributing, consider giving more ideas in forms of issues or maybe discuss on the issues how to solve them (or maybe even do a pull request to solve some of the issues).
I hope that you have enjoyed (or were at least helped by) this tutorial. Feel free to leave feedback for improvements or new information!
Imagine you have a (very contrived) language as shown below (i.e. some root that contains children of an abstract concept named A and Bs and Cs extend A):
By default, all As and Bs can be substituted (Ctrl/Cmd+Space) in the nlist of MyRoot:
Now, if we want to restrict this, to, let’s say, only Bs, then we can use a ‘can be parent’ constraint on MyRoot that looks like this:
Note that you can’t use childNode.isInstanceOf(B), since this would only work when the childNode is already instantiated (so it would give you an empty completion menu as an uninstantiated child node is not and will never be an instance of B.
The resulting completion in the editor looks like this:
This type of construction is especially useful if you make a language which extends another language that you don’t want to change, but of which you want to restrict certain concepts in your extension.
An example of this (currently for MPS2018.1.5) can be found here: https://github.com/DSLFoundry/mps-examples/tree/master/ChildConstraints
Between MPS major versions, the MPS global caches are nicely separated (e.g. for MPS 2017.1, the cache directory is $HOME/.MPS2017.1 and for MPS 2017.2, the cache directory is $HOME/.MPS2017.2, where $HOME is your home directory, e.g. C:\Users\<yourusername> on Windows or /home/<yourusername> on Linux). Sometimes, there are reasons why you would want to change the cache for a specific MPS distribution on your filesystem. Some of these reasons are:
- Run different distributions or RCPs of MPS that have the same major version at the same time on your machine
- Separate two MPS global caches (e.g. a cache for production in some project you work on and another cache for a course you have to give tomorrow), so they don’t interfere with each other.
- Be able to open two modules (language or solution) with the same name/ID, without the two getting intermixed in the global cache (or MPS thinking that one has already been loaded, making the other read-only).
By default, the MPS cache-directory for an MPS major version called <version> (located under your $HOME directory) is .MPS<version>, e.g.: .MPS2017.2.
You can change this default location by adding a line like this in your vmoptions (for Windows, it’s located under your <mps installdir>\bin\mps64.exe.vmoptions):
This would lead to the MPS that you run from <mps installdir> to maintain a cache directory called .MPS-MYOWNTHING in your $HOME directory.
The mbeddr RCP is great, because it gives you a focused MPS-based IDE, including mbeddr.platform, which you can use to write models in the embedded software domain.
However, I find it often useful to have the mbeddr.platform plugins (which are generic pieces of functionality useful for any DSL development) available for development as a language engineer, not as someone who wants to write mbeddr models.
If you want the full-blown mbeddr development environment (which requires you to do a build of the entire mbeddr.platform and mbeddr DSL stack), just follow the mbeddr build environment setup howto.
However, I often have the use case where I just quickly want to setup an MPS distribution (e.g. for student courses or student projects) with the mbeddr DSLs included. In this case I just perform an easy procedure:
- Look on the mbeddr.core github releases page for the latest nightly build
- Determine which MPS major and minor versions are used by the latest nightly build:
- For a specific release on the releases page, download the source code (e.g. zip file)
- Look in the upper subdirectory of the zip for the file called build.gradle. There is a line in this file that starts with ext.mpsBuild =, e.g.: ext.mpsBuild = “2017.3.4”. The version number is the specific MPS build that this mbeddr nightly is using.
- In the same build.gradle file, there is a line that starts with ext.mpsMajor =, e.g. ext.mpsMajor = “2017.3”. This is the MPS major version used.
- Download the unified package (zip-file) of the MPS version you found in build.gradle. Currently the MPS download links are nicely formatted like this: https://download.jetbrains.com/mps/<mpsMajor>/MPS-<mpsBuild>.zip.
So, e.g. for MPS 2017.3.4 you can manually construct the download URL (which will be https://download.jetbrains.com/mps/2017.3/MPS-2017.3.4.zip) and then use your favorite download tool (e.g. wget or your browser) to download it.
- From the mbeddr releases page, download, depending on what you need:
- com.mbeddr.allInOne.zip (full mbeddr distribution including mbeddr.platform and embedded development DSLs)
- platform-distribution.zip (mbeddr.platform only).
- Unzip the MPS zip (extract here, this will create a subfolder called MPS <mpsMajor>, e.g. MPS 2017.3) and then unzip the chosen mbeddr zip (either com.mbeddr.allInOne.zip or platform-distribution.zip) into the MPS folder (which places all the mbeddr plugins into the plugins subdirectory of your MPS folder).
- Inside your MPS folder, copy all the files from bin/<platform> (where <platform> is the platform you work on: win, Linux, or mac) one directory up (so the executables for your platform end up in the bin directory).
- Either set a command line variable called MPS_JDK which points to your Java JDK installation, or edit your relevant launcher file (e.g. for Windows this is bin/mps.bat) and locate the line where JDK= is set and put your JDK path there).
Additional info on 25 July 2018: JetBrains nowadays uses a self-maintained JDK for MPS. So far I haven’t been able to find it on the JetBrains site, but Itemis provides downloads of these JDKs for all three platforms (Linux, Windows, Mac) at https://projects.itemis.de/nexus/, e.g.:
Additional info on 29 July 2018: Instead of unzipping mbeddr plugins into MPS, you can also just install a native version of MPS (using setup.exe or the Linux or Mac installation package) and then use global libraries to point to the mbeddr plugins: