A brief update on how the artifacts of mbeddr, mbeddr.platform, and IETS3 have been changed (for the people that have to deal with older MPS models made using one of the mentioned extension packages): up until MPS 2019.2, it was possible to download all artifacts from build.mbeddr.com. Today, I was trying to do this as well, but it seems all the artifacts have been wiped for mentioned MPS versions (maybe some space needed to be made free on build.mbeddr.com, or maybe some internal build process changed). Now you have to actually get the label of a version that has successfully built from build.mbeddr.com (e.g. 2019.3.22698.f4cbbf7 of the mbeddr platform or 2020.1.4698.e240d62 of IETS3.opensource, the latter leading to https://projects.itemis.de/nexus/content/repositories/mbeddr/org/iets3/opensource/2020.1.4698.e240d62/opensource-2020.1.4698.e240d62.zip) and then get the corresponding artifacts from https://projects.itemis.de/nexus/content/repositories/mbeddr. Especially for IETS3 (KernelF), this is really needed.
Note: since MPS 2020.2, it is possible to simply get release artifacts from github (e.g. https://github.com/IETS3/iets3.opensource/releases/download/nightly-2020.2.4726.e49ca16/org.iets3.opensource-with-dependencies-2020.2.4726.e49ca16.zip). And these artifacts even include the dependencies (such as mbeddr platform, which is needed by KernelF/IETS3).
Like most programs, MPS writes messages to a log file — lots of messages. To make sense of so many messages, it helps to use a tool that improves the visual presentation, for example:
grid display: separate fields and align them nicely, like timestamp, logger, message
color coding for message severity, like bright red for ERROR
set column order: move a column out of view if it is not relevant to you.
filter out messages, like: block all messages from logger com.microsoft.alm.
Apache Chainsaw is such a tool.
Why choose Apache Chainsaw? First of all it is free. Second, it is a companion to the log4j library that is used by MPS. Chainsaw is designed to play nice with log4j. Third, it has been around for a long time, so it can be expected to offer relevant functionality. As a disclaimer: I myself have not used chainsaw before.
How to configure Chainsaw? Let’s make a choice here: we are going to connect Chainsaw to the MPS log via the file system, not via sockets. Sure, the log4j architecture offers a socket based approach, but that is much more involved. It would require configuration changes at the MPS side of things, whereas using the log file only requires a proper configuration at the Chainsaw side. Moreover, changing the configuration at the MPS side has the difficulty that the log4j version that MPS uses is way beyond end-of-life, making it difficult to find the proper components and documentation. MPS still uses log4j version 1.2.17 released Jan 2013 and log4j version 1 was declared end of life in 2015. I briefly tried to get the sockets running anyway but I did not succeed. Maybe at a later time.
It proved not to be very difficult to get Chainsaw to connect to the MPS log file. First, a short explanation if you want quick results. Second, a longer explanation if you want to know how this works under the hood. Finally the details if you want to explore further.
HOWTO – the short explanation
This XML snippet tells Chainsaw where to find the MPS log file and how to parse it.
Adapt the ‘fileURL’ parameter to your situation: 1/ change the user name “jennek” to yours 2/ change the MPS version “2019.1” to yours. Check if you can find the log file at this path. If it is not there, please read the longer explanation
Open the default configuration file of Chainsaw and add the XML snippet. This file is named “receiver-config.xml” and it should be in folder “.chainsaw ” in your home directory. If it is not there, please read the longer explanation.
Restart Chainsaw. You should now see a tab named file-/C:/Users/jennek/.MPS2019.1/system/log/idea.log. Tweak the view to your liking using the menu item “Tab Preferences” under the “Current Tab” menu. Your tweaks are persisted by Chainsaw.
One more thing. In the logger column, you may see weird entries like m.microsoft.alm.plugin.operations.BuildStatusLookupOperation. This is caused by MPS using a 60 character limitation on the logger name, truncating com.microsoft to m.microsoft. This results in quite a few spurious loggers in the Logger Tree Pane on the left. If you want to fix this, do this:
Open <MPS-home>/bin/log.xml in your favorite editor
Find the appender element with <param name="file" value="$LOG_DIR$/idea.log"/>
Just below the line find this line <param name="ConversionPattern" value="%d [%7r] %6p - %30.30c - %m \n"/>
Remove the 30.30 which means: “minimum field width 30, maximum field width 30 more”, and be sure to keep the %c
Now MPS will write the logger name to the log file without truncation.
Under the Hood – the longer explanation
To understand how Chainsaw should read the log file, let us look at how MPS writes the log file. As said, MPS uses log4j version 1. The log4j configuration is typically found in a file named log4j.xml but in the MPS distribution you will not find this file. Instead MPS has a file named log.xml in the /bin directory that is very similar to a log4j.xml file. This file contains a number of path variables that need to be expanded. Only after expansion the log.xml file is passed to log4j. To see this in action, read the gory details down below.
This is the part that tells log4j to append log messages using a rolling file mechanism. The file parameter with value $LOG_DIR$/idea.log says to which file to append– where the $LOG_DIR$ is expanded. The ConversionPattern tells that the log message must be formatted in a specific way. This is precisely the information that Chainsaw needs to find the file and to split the log line into separate fields.
Here Chainsaw makes life easy for us. It would be a burden to first look at the conversion pattern of the appender that combines the message fields into a line and then to construct the inverse pattern to unravel this line into separate fields. Chainsaw will do this for us. Try this in Chainsaw (see screenshot below):
Pick menu item “Load Chainsaw Configuration” under menu “File”
Select “Use fileappender entries from a log4j config file”
Press”Open File…” and navigate to the log.xml file in the bin folder of your MPS install
Press “Save configuration as…” and navigate to some temporary folder and save as -say- mps.xml
Open mps. xml in your favorite editor and there you have it: Chainsaw has written the receivers (plugin) for the senders (appenders) in log.xml. The plugin we need is the one with “$LOG_DIR$/idea.log” in the element with name “fileURL”. See below.
But before we can use it we need to fix a few things
in the element with the name “fileURL”, replace the path with $LOG_DIR$ with the path on your system. See the HOWTO section above.
in the element with name “logFormat”, delete the ” ” — this is a bug. If you keep it, the splitting of a log line into fields fails and the entire log line ends up in the message column in Chainsaw
After these edits you should have an XML snippet at the beginning of this post. This snippet you may include in a Chainsaw configuration file — either the one that is loaded when Chainsaw starts up or one that you load on demand. To find the start-up configuration of Chainsaw, do this:
Pick menu item “Show Application-wide Preferences…” under menu “View”
use the path in the “Auto Config URL” under the tab “General”, typically file:/C:/Users/<username>/.chainsaw/receiver-config.xml
Logging log4j itself – for further exploration
If you want to explore how the log4j system is behaving, try this:
find the file that launches MPS, on Windows this is <MPS-home>/bin/mps.bat
where you see javaw.exe, remove the w and keep the java.exe.
This keeps the console window open while running MPS, showing all the logging from MPS. However this may slow down the IDE considerably. This is one good reason to view the log with Chainsaw instead. Now when you start up MPS you see that the very first lines are:
Markus Völter has written a really nice Medium article about making the core of the functionality of a system (he states software system, but I think this can be generalized to system) where the domain is concerned explicit. He calls this embodiment of the accrued expertise of a company (or a person) Fachlichkeit.
Since this is a word that is hard to pronounce, but really important, I have spent some time with Eugen Schindler (@eugenschindler) to find an English term that, in my opinion, does cover the meaning well enough and also enables you to make it into a verb and a verbal noun.
The word is: acumen. Using this word in the context described by Markus, would look like this:
Acumen (noun): the ability to make good judgements and take quick decisions
A domain expert has domain knowledge and acumen (due to this domain knowledge).
Acuate (verb): to make pungent or sharp
A domain expert’s knowledge can be acuated (i.e. explain, judge, analyze, decide) into some form (i.e. from writing a document to encoding it into a DSL structure/meta-model) for a purpose (i.e. solving specific problems)
Knowledge = WHAT, form = HOW, purpose = WHY, WHO? (i.e. language engineer), …
Let me tell you about something awesome that I have been working on with Mike Vlassiev from the JetBrains people (or rather, they have been working on it and I have done testing and provided feedback): the MPS Extensions (https://github.com/JetBrains/MPS-extensions) are now automatically made available on the JetBrains marketplace on every new MPS and MPS Extensions release.
Why is that cool? Well, the MPS Extensions, which are maintened by the MPS Community and curated by Itemis (https://www.itemis.com/de/), are very useful plugins which make language developers even more productive than an out-of-the-box installed MPS can do.
Some of the most prominent examples in the MPS Extensions are:
Diagrammatic, tabular, and math notation for MPS languages (de.itemis.mps.editor.diagram, de.slisson.mps.tables, and de.itemis.mps.editor.math)
Incrementally maintained non-editable models from existing models based on model-to-model transformations (de.q60.mps.shadowmodels)
Template-based text generation (com.dslfoundry.plaintextgen)
And now all these goodies have just been made much more accessible to novice users of MPS via the JetBrains marketplace. This also means, that dependencies for plugins are automatically resolved, which is important for MPS Extensions, since the plugins there are built with a high level of re-usability in mind (and hence dependencies over plugins exist).
I think this is a very important step in the direction of getting the results of the community accessible to new users. A next good step will be to make all the MPS Extensions directly available via one meta-package.
The com.dslfoundry.plaintextgen extension (https://jetbrains.github.io/MPS-extensions/extensions/plaintext-gen/) for JetBrains MPS (https://www.jetbrains.com/mps/) enables a language designer to write model-to-text generation templates in pretty much the same way one writes model-to-model transformations in MPS. This is very useful for many reasons, the most prominent two of which are in my opinion: you don’t have or don’t want to build a generation target language for your text generation needs (e.g. because you need to make some model-to-text transformation very quickly) or if you want to work more like classical template-based text generation similar to Xtext.
This tutorial describes some of the most common uses of the plaintextgen.
This tutorial assumes that you have basic knowledge of MPS, i.e. how to create a new project and how to make a language with an editor and a generator.
How to use plaintextgen?
Before you can use plaintextgen, you need to install it first. You can either do so by downloading the MPS Extensions (https://github.com/JetBrains/MPS-extensions/releases) and copying them into your <MPS_Installation_Folder>\plugins folder, adding them to your global libraries (File –> Settings –> Build, Execution, Deployment –> Global Libraries), or by installing the com.dslfoundry.plaintextgen plugin from the MPS Marketplace (File –> Settings –> Plugins –> Marketplace).
Once you have com.dslfoundry.plaintextgen installed, you can use it within a generator template. The idea behind plaintextgen is to reproduce the look-and-feel of the textual editor aspect as much as possible, since both text generator and textual editor in principle handle the rendering of concrete syntax for a textual notation on your model. Therefore, I am not going to explain each detail and you can try to use the basis of your mental model of textual editors (e.g. vertical and horizontal collections, newlines, surrounding blocks with collections) on the plaintextgen language. Names are a bit different, e.g. vertical collection (from editor) versus VerticalLines, horizontal collection versus Line, CellModel versus Word, etc. You can see the metamodel of plaintextgen in the structure of the language.
If you feel that certain things are very different from the editors aspect, then this is most probably a bug or not-yet-implemented feature. Feel free to file an issue (or even better: make a pull request) on https://github.com/JetBrains/MPS-extensions for the plaintextgen component.
I will illustrate most of plaintextgen’s basic functionality by extending the generator of the example language com.dslfoundry.plaintextgen.example.testlang. What I will do is add a generator for a C++ file, based on an existing C++ listing from the ThinkC++ book by Allen B. Downey ( http://greenteapress.com/thinkcpp/index.html ). The listing in question is one of the accompanying code samples on github: https://github.com/AllenDowney/ThinkCPP/blob/master/code/cha11/cha11.cpp . The templating of the code is only to illustrate the mechanism and will probably be completely nonsensical, so don’t try to think too hard on what the end result would mean or whether it can be compiled 🙂 Now, if you have any suggestions for improvements, I’ll be happy to hear that in the comments.
If you want to reproduce this tutorial based on com.dslfoundry.plaintextgen.example.testlang, you can clone the https://github.com/JetBrains/MPS-extensions repository and open the MPS project under the code directory. The MPS version that is supported on master can be found at the key ext.mpsMajor in the file build.gradle in the root of the repository. If you need the code for an older version, there are maintenance/mps<version> branches. States of the branches are described at https://jetbrains.github.io/MPS-extensions/#current-versions. One last thing to add before we begin: I will add MPS links that you can click from here if you have the code project of the MPS Extensions repository open and it will open the specific node that the link points to.
So let’s get going!
Just for reference, com.dslfoundry.plaintextgen.example.testlang can be found here in the project:
First we will have a brief look at the test model (instance of the TestConcept concept) in com.dslfoundry.plaintextgen.testlang.sandbox:
So we have a TestConcept instance which has a property called testProperty with value TestValue. Then, it also has two children in its list of testSubConcepts with (respectively) testProperty values TestValue2 and TestValue3.
Let’s make our first change to the language by adding a new plaintextgen-based root template to its generator:
So far nothing special, it’s just the creation of a root template. The only thing you need to know is to import the com.dslfoundry.plaintextgen language (Ctrl+L+L and search for the language, then select it and press enter or double click it).
So we only need to select the plain text from the C++ source file and then do execute the Paste Unstructured Text on the plaintextgen root template. Plaintextgen takes care of analyzing indentations and makes a full plaintextgen template for you.
Next, let’s feed the testProperty of the root TestConcept into the template:
This works again pretty much the same as the normal MPS model-to-model generator templating mechanism. We just add a property macro on the filename based on the testProperty value of the root TestConcept and then we add a filename extension .cpp. Then we run a build and perform a Preview Generated Text action (via right-click) on the model root. The result is a text file called TestValue.cpp which lists the hard-coded contents that we have imported in the last step.
To continue, we will feed the value of testProperty of the root TestConcept to the struct name:
Here we use the Split-extract Selection into Word intention of plaintextgen in order to make it a separate word that can be templated. Then we put a node macro for testProperty on the split-extracted word. When we build this and then again run a Preview Generated Text action, we see that the name Time has been replaced by TestValue, as would be expected.
As a next step, we will feed the testProperty values of the testSubConcepts children ( TestValue2 and TestValue3 in the model) to the struct members:
To do this, first we delete all but one of the struct members from the template. Now, we apply a LOOP macro on the template (again, just normal MPS generator macro stuff); N.B.: don’t forget to include the newline, or else you will miss newlines in the output. We finish this step off by doing a split-extract on the name of the member and feeding the testProperty value to it via a property macro. When we build this, the output is as expected: we have two members called TestValue2 and TestValue3.
Finally, we are going to add an implementation for each member based on the testProperty names of the testSubConcepts children:
Again, we delete all implementation functions except for one. Since we cannot apply a macro on multiple nodes, we need to wrap the multiple lines of the implementation function into one node. To do this, we use the Surround with Vertical Collection selection intention (N.B.: we did not press the light bulb, but pressed Alt+Enter to trigger the intention since it’s a special intention which can be executed on a selection of multiple nodes, provided by the language de.itemis.mps.selection.intentions). Now that the multiple lines are wrapped, we can apply a LOOP macro, do again the split-extract on the name of the implementation, and use a property macro to feed the value of the testProperty of each of the testSubConcepts children.
Of course, we could continue and really make this a much more refined example, but this should be enough for now to illustrate the major mode of working with plaintextgen to do your own templates.
A last remark: this tutorial took a template imported from plain text (a C++ file) as a basis and did some plaintextgen operations on it to get to the desired results. Of course, you can also build your own plaintextgen templates from scratch (adding new word, vertical lines, lines, etc.), but if you have a textual source, then I think the above method is much easier.
Often times, when creating a new language, I skip the “Create Sandbox Solution” checkbox. Recently, I have learned that the sandboxes are actually not just solutions with the name sandbox. Because of their property of actually residing in the language folder itself, many MPS Extensions are actually using sandboxes to package example models with a language to show its use.
Now imagine that you have created a language without a sandbox, then you may wonder how to add one. Although the answer is very simple, I still describe it here, since it may not be immediately obvious.
I’ll following the com.jetbrains.etc naming convention in this example. Imagine you have a language called com.dslfoundry.mylanguage and which doesn’t have a sandbox yet. To add a sandbox to it:
Create a new solution called com.dslfoundry.mylanguage.sandbox
Add a new model to it called com.dslfoundry.mylanguage.sandbox (so normally, you just do an “add model” and then remove the dot at the end of com.dslfoundry.mylanguage.sandbox.)
Open a file explorer and navigate to the folder of your language (usually that is <project_folder>/languages/com.dslfoundry.mylanguage)
Move the entire folder of the earlier created com.dslfoundry.sandbox solution to your language folder
Now rename com.dslfoundry.sandbox to just sandbox
To make the sandbox show up on your project, you have to right-click your project in the logical view and select properties, then click the + to add the .msd-file of the sandbox solution.
MPS comes with pretty good default settings, but there are some settings that I really like to change as soon as I have a fresh MPS IDE in front of my nose. Below is a description of such settings.
Hide empty aspect tabs
By default, MPS always shows all tabs for all aspects (also aspects that are empty) for each concept. I have found that I keep more overview when the empty aspects of a concept are hidden. You can change this, by setting the Aspect Tabs option to Each aspect node in a separate tab:
Hide generator warnings that you cannot fix
So far (right up until MPS 2019.1), MPS shows some really annoying “Failed to replace dynamic reference” generator warnings. They are annoying because they pollute the build logging and you cannot do anything about them as a language engineer. You can remove these (without being afraid that your languages will not be checked when you commit, because MPS checks all modified languages and models on every commit) by unchecking Show warnings in the generator settings:
Disable language checking in every build
When working incrementally on a language implementation, I often run many rebuilds with every small delta that I want to test for my language. Since model/language checking is not incremental, having a check every time when I need to build a language can take a lot of cumulatively collected overhead time. Therefore, I like to, by default, turn off the model checking for every build.
There is a caveat, though: since MPS’ language development workflow currently relies on languages/models being checked in the IDE and not in command line/buildserver build, you should do model checking before checking in! I still change the default setting, which requires more discipline from me, but saves me some time.
Disabling continous modelchecking can be accomplished by unchecking the item Build –> Check models before generation item in the menu:
Prevent auto-opening of last-opened project
For me, the reason of shutting down or restarting MPS is usually because I need to switch between projects or because I need to play with certain settings. This makes the default feature of reloading your last opened project upon MPS startup not very useful. I turn it off here:
This piece of information is provided thanks to Hristina Moneva (see coments).
The memory indicator does not only give you a way to see how much memory is allocated to the VM that MPS and your memory use, but it also gives you a way to do garbage collection manually. It is located in the lower right corner and looks like this:
Adding the memory indicator can also be done via the settings menu:
The mps-plaintextgen plugin has been upgraded with new functionality for fine-grained control of whitespaces. Such functionality is very useful for languages that are fiddly with whitespace such as reStructuredText or where you just want to have very fine control (such as in JSON or in some instances Python).
Plaintextflow is part of the plaintextgen plugin, but can be imported as a separate language, so existing functionality of plaintextgen is not disturbed.
For many purposes of end-users and language engineers , the standard MPS mechanism with tooltips and Event Log is sufficient:
However, in some cases more information is needed. This is where the MPS log files come into play. There are two ways to reach this log:
You can use Help –> Show Log in Explorer (or the Linux/Mac equivalent of this) from the MPS main menu. This will point you to the rotating idea.log (usually located in your $HOME/.MPS<version>/system/log directory). In this case you will either inspect the log using your favorite text editor, or you can use a dedicated log4j-compatible log viewer (just google for it).
You can look back at the command line from where you started MPS. In case of Linux (and I think Mac), this will go automatically, if you start MPS from a console using mps.sh (which is located in the bin directory of your MPS installation), but in case of Windows, you have to adapt your mps.bat (which is also located in the bin directory of your MPS installation) as follows: replace all the strings that say “javaw” with “java” (currently there are two strings; this will ensure that there is a console open next to your MPS instance into which you can look in real-time).