SharpDevelop Community

SharpDevelop 5 - NRefactory 5 + semantic highlighting

While we have several more releases of SharpDevelop 4.x planned; we are also in parallel working on a big new release: SharpDevelop 5.0.

The major change in SharpDevelop 5 will be the complete rewrite of the NRefactory and SharpDevelop.Dom libraries, which together implement code-completion and many refactorings. Work on NRefactory 5 originally started in August 2010, as a new library written from scratch.

The NRefactory C# parser and abstract syntax tree was rewritten by Mike Krüger. The new version of the AST contains all individual tokens, and has positions on every node - this makes the implementation of refactorings a lot easier.

The new AST is already in use in ILSpy - when we started working on ILSpy in February, we immediately ported the old decompiler (from David's dissertation) to NRefactory 5. As part of the ILSpy development, I added several new features to NRefactory - for example the pattern matching support for the AST, and the visitor that converts the AST back to C# code.

I also wrote a new resolver for NRefactory (the resolver previously was part of SD.Dom). The main goals here were to:

Improve performance by making the step of resolving type references an explicit method call

Follow the C# specification more closely

Improve performance in general

Add support for projects that target framework X but get used in framework Y projects (ITypeResolveContext).

I started working on the new resolver in October 2010 and completed it in August 2011 (feature-complete for C# 4.0). All this work was besides the work on SD 4.x and ILSpy, not to mention my regular "job" (I'm a student and work on SharpDevelop in my free time).

Since then I've started integrating NRefactory 5 into SharpDevelop. This still is a huge task, as basically anything that deals with C# code, - i.e. class browser, code completion, go to definition, find references, all refactorings, the forms designer loader, etc. - needs to be rewritten or at least adjusted.

But today I tried something a little different: a new feature - semantic highlighting.

This code (with syntax highlighting) was copied out of an early SharpDevelop 5 alpha version. As you can see, we now highlight references to types in a blue-greenish color - but only when semantics of the code actually refer to the type, as the difference between the static and instance method calls shows.

Fields are highlighted in italics, which allows telling fields and local variables apart in a large method. Additionally, the dark blue/bold method highlighting is now only applied when the method exists - not to delegate-typed fields, and not to missing methods. And context-dependent keywords like value are highlighted only when used in the appropriate context.

The highlighting engine has the full power of the NRefactory resolver available, which allows use to easily experiment with new types of highlightings - for example, it would take only three lines of code to, say, highlight extension methods in a different color than normal methods. Or value types differently than reference types.

We also plan to detect syntax errors and the most common semantic errors (method not found, missing parameter, cannot convert from type X to Y) and highlight those errors while typing, so that you don't have to recompile as often.

There are tons of potential highlightings that give useful information - but if we use all of them, the text editor will look way too colorful and noisy. What do you think is most important?

In my experience what is important to me is not important to the next person. I like to highlight certain things over others. I think this has to do partly with how different people perceive the world. The best way would be to expose as many of these options as possible into a settings page where if I want I can go nuts and assign as many colors as I want and then pick a reasonable default subset to color. Leave the rest set to "normal" whatever that is.

I do like that you have more precision than most other systems as you aren't simply using a syntactic parser. Differentiating value is cool.

There isn't much documentation yet, only the README and a demo application that can parse code and then displays the AST in a TreeView.

The new AST is for C# only.

We gave up on the idea of using a shared AST for C# and VB. The two languages use different syntax for the same constructs, but also have slightly different semantics for those same constructs - it turned out that having a shared AST did not lead to much code reuse, but added a lot of complexity - pretty much any code consuming the AST either had lots of 'if (isVisualBasic)' special cases, or was incorrect for ASTs produced the VB parser.

The type system is shared by all languages, and is closely following the .NET type system - so any language compiling to .NET can be represented in it, but it might need some tricks similar to the ones employed by the compiler (e.g. global methods must be stored in a class).

Could Microsoft Roslyn (blogs.msdn.com/.../the-roslyn-preview-is-now-available.aspx) become relevant for this kind of thing in the future? It seems like Roslyn would enable much of the same functionality, with 100% identical semantics to the real compiler, full VB support, and no parsing, AST, etc work for you guys.