Author: Ken Domino

Ken is a software engineer and has worked professionally in the field since 1984. He has a masters of science in computer engineering and a masters of science in biology. For more information, please see http://kennethdomino.com.

Surprise! I’ve been implementing something like the Language Server Protocol (LSP) with AntlrVSIX. The LSP is an abstraction for an editor (ed) using JSON as the lingua franca between the editor and a GUI client. The editor server: offers the persistence of code files; organizes files in a “workspace”; provides an API for edits, tagging, go to def, find all refs, reformat, code completion, defs for tooltips, etc. LSP is a step in the right direction because it separates an editor server backend from a GUI frontend, which is what AntlrVSIX is about.

After a lot of hemming and hawing, I am now adding the concept of “workspaces” to AntlrVSIX. By this I mean an equivalent of workspaces that is defined in Roslyn. In AntlrVSIX, a Document will be a source file; a Project will be a collection of Documents; a program will be a Workspace, a collection of Projects. Properties on a Project or Document are copied to the equivalent AntlrVSIX object as a property list.

I’m now working on adding in a full symbol table implementation into AntlrVSIX. This will help make the extension much more powerful with tasks such as tagging and navigating to defs.

For better or worse, I’m starting with the implementation in Antlr Symtab. It looks like it’ll work out pretty well, but it is missing certain classes, and needs a little polishing. For example, VariableSymbol is the base class for FieldSymbol and ParameterSymbol, but there isn’t a class for variables defined and referenced in a method body. This makes it somewhat difficult to distinguish between a field and a local variable in a block.

Also, for better or worse, I’ve added Mono.Cecil into the symbol table code. I’m not sure whether I’ll need this, but it’s there just in case. Mono contains a reader of PE files, which might be useful. But, what I’m really thinking of is the equivalent over in Java and other languages.

(1) I’m adding a Rust parser to AntlrVSIX. The grammar I’m using is an old grammar from Jason Orendorff. Unfortunately, it hasn’t been updated for several years. Also, it looks like it isn’t correct because there are cases where tokens are not correctly defined, e.g., ‘>’ ‘=’ instead of ‘>=’. It’s incorrect because the parser would allow a space between the ‘>’ and ‘=’ in the input, which is probably not intended. I’m first updating the grammar by separating the parser and lexer grammars, then I’ll add in rules recognized by another grammar.

(2) I’m really disappointed in the fact that no one seems to have written a general wrapper for command-line tools for Msbuild. I wrote a wrapper for the Antlr Java tool, but I’ve noticed that I’m going to have to write yet again another wrapper for Rust’s tools. I now realize that all of these wrappers can be generalized into a single wrapper with a specification of inputs and outputs. When I’m finished with the Rust grammar, I plan to write this tool, and rewrite the Antlr Java wrapper with it. It is no wonder why it seems everyone is using Visual Studio Code. I would too, but it’s written is the most awful of all languages, Javascript.

As I pointed out before, it now seems possible to write a VS extension that supports any programming language. I’ve been updating AntlrVSIX to input a description of the syntax of a programming language and tag a file of that language in the editor. I have this now working for Antlr and Java, and plan to try a dozen or so languages.

At issue is identifying what are the commonalities of a programming language, such as defining and applied occurrences, and how the editor should use that.

However, some extensions tag the programming language that my extension is also trying to tag, which results in multiple competing and conflicting classifiers and formatters. I solved this problem by changing the IContentType associated with a buffer to one that I define for the extension. But, ideally, I’d prefer to just “turn off” any formatters that are competing with my formatter.

It may not look like much, but here is how Java is formatted by Visual Studio by default, and how my extension formats it (first attempt to get something working).

Current VS 2019 “code++.java” formatting. Variable “token” labeled erroneously in comments.Formatting from the Antlr extension. Squiggles indicate no defining occurrence in the file, but should be found via imported libraries.

Now that I have my AntlrVSIX extension working for Visual Studio IDE, I have now set my sights on a “meta-language” editor. This picks up the idea of an add-in that supports Antlr, and brings it forward to an add-in that supports any language. For this to work, the grammar for the language would be specified in Antlr. I would also need to add in information on what parts of the parse tree constitute what classified types for tagging in the editor. A corpus would be needed for Codebuff to work on reformatting a file containing code in the language. All seems doable. Or is it?

Due to my work on Piggy and Campy, I’m extending my Antlr4 extension for Visual Studio in a number of ways. The plug-in hasn’t been updated for two years, and there are no extensions for Antlr that work with Visual Studio 2019, so it is due for an update. In fact, there isn’t a single extension for Antlr for Visual Studio 2019, and Antlr.org removed Visual Studio from the list of developer tools for Antlr.

Changes

Targeting Visual Studio 2019 and 2017

I’m not sure where I read it, but “good practice” says I should only support the current version and one version previous.

Improved Tagging

I’ve added new tags for channels and modes, and the whole classification and tagging routines now check if the source has changed before reparsing, which improves performance.

A menu for AntlrVSIX in Extensions

In Visual Studio 2019, the menus for extensions are where you should look–under “Extensions”. Previously, I have to admit that it was hard to figure out the UI for AntlrVSIX, hidden under various menus, or available through a right-click context menu. That’s basically bad UI design.

Although the right-click context menu for AntlrVSIX is still available, it mirrors exactly what you see in the main menu under “Extensions -> AntlrVSIX”.

Navigation to next and previous rules in a grammar file

I’m a big fan of the old-style Emacs and even older Vi! Back then, UI was keyboard-oriented. It was much more efficient instead of moving around this stupid mouse, pointing to something I can hardly see on the screen.

I’ve added a few new ways to navigate around a grammar file with some shortcuts. “Next rule” jumps to the next rule in a grammar; “Previous rule” jumps to the previous rule in the grammar; “Go to Visitor” navigates to a visitor method for a tree node corresponding to a non-terminal.

Generation of Visitors and Listeners

Go to visitor/listener jumps to C# code for grammar symbol. If the method doesn’t exist, then the Visitor or Listener classes and methods are generated.

Options for the extension

An options dialog box for extension-specific parameters is now included. It is needed for things like where to find the “corpus” of examples of formatted grammars that Codebuff will use to format your grammar.

Performance

While AntlrVSIX is the only extension for VS 2019, it’s rarely used compared to several other older–and in my opinion, less useful extensions. AntlrVSIX has had only one review, which says the performance sucks. (Thanks, dude for not saying *what* specifically are you trying to do that is slow. I’d never have you on my QA team–not helpful descriptions).

Performance has taken a more front seat with this release. Before parsing, I check whether the code buffer has changed.

I spent a lot of the time cleaning up the source code. At the time when I first wrote the extension, I had no clue how to modify which feature in the UI because the documentation for Visual Studio extensions is absolutely terrible. I now have a better command of what is going on and fixed a lot of the code.

Migration from packages.config to PackageReference builds

It’s absolutely appalling that the examples for extensions Microsoft provides is out of date and are so poor. Many still use the “packages.config” file to list the NuGet dependencies of a C# project. I’ve updated AntrlVSIX to use the latest format, which required the tool to be migrated to Net 4.7.2.

You can’t really have an extension for a language like Antlr if you don’t include some way of actually *building* the project that uses the language. After first writing AntlrVSIX, I then modified Antlr4cs, a wrapper for a C# version of the Antlr tool, to just be a wrapper the standard Antlr4 Java tool. So, it now generates the parsers/lexers in the IDE.

In addition, the Antlr4BuildTasks had a bug where one couldn’t change the build mode of the .g4 file to none. This is required for multi-file grammars that use the “import” statement in Antlr.

Fixing Intellisense

I fixed the Intellisense for the extension. The tooltip now works when hovering over a single-character grammar symbol. Command completion now offers a list of grammar symbols in the grammar.

*****

The work on the extension has been going on for two weeks and should be finished in the next week.

I was recently was trying to use my Visual Studio extension for Antlr in Visual Studio 2019, when I found that it just isn’t working anymore. In fact, I couldn’t even install the extension because it wouldn’t even show up in the search for the plug-in. In fact, there weren’t any extensions available for “Antlr” for Visual Studio 2019! I guess I hadn’t ported the add-in to VS 2019, so I decided to work on that.

Up to now, I have been making an assumption for the Piggy transformation system that regular expressions and NFA recognizers can be used to match arbitrary ASTs. After all, a lot of research has been published for tree recognizers, including XPath, TreeRegex, and Tregex, which are NFA based. With all this, I lost sight of the problem of trying to match subtrees within an AST. Unfortunately, pattern matching of ASTs is not regular, and so cannot be recognized by an NFA. This blog entry goes over why NFAs cannot be used, and how Piggy patterns should work.

1/2: Adding to #Antlrvsix an analysis tool of #Antlr grammars. This is how it works with cycle detection and useless lexer rules. There are some issues in the responsiveness of the MS LSP client for VS2019.

Adding to #Antlrvsix the refactoring to remove useless parentheses in an #Antlr grammar. This is how it works with the extra parentheses in the arrayAccess_lf_primary rule in Java9.g4 that nobody knew were there. Only yet starting to scratch the surface of grammar optimizations.

Implementing #Antlr grammar fold refactorings in #Antlrvsix. Two types: extract a selected sequence of symbols and make a rule (shown first); replace all occurrences in the grammar with a folded rule (shown second). Spacing and comments do not matter.