Firstly, I’ve just pushed a new, 2.0.0 ALPHA pre-release version of the xUnit.net runner for ReSharper to the extension gallery. It adds support for xUnit.net 2.0 beta 2. You can get it from ReSharper’s Extension Manager. Make sure you select the “pre-release” drop down in the Extension Manager first.

Secondly, the project is no longer hosted on CodePlex, but has moved to GitHub, and lives under the xunit org. All issues and pull requests should go there please.

xUnit.net 2

You should consider this an alpha version, because, well, it is. It works, I’ve got a bunch of tests to prove it, but hey, it’s all new and different, so if you find any issues, please, please, please, let me know (twitter or, even better, GitHub. Did I mention the project’s moved?) In fact, any kind of feedback is appreciated, good or bad.

Thanks to the excellent support built into xunit2 by Brad and Jim, the new runner can handle both your xunit1 and xunit2 tests. Everything works as you’d expect – tests are highlighted in the editor, and you can run and debug them as normal.

However, this is an alpha build. There are couple of important caveats:

Parallelisation is DISABLED in this release. This is due to how xunit2 handles errors. Currently, there is no way for me to gracefully handle errors thrown when a class or collection fixture throws an exception. Hopefully this can be resolved, and I’ll enable parallelisation.

Test discovery in the editor is currently handled by xunit1. This needs migration, but is a complex beast. It means that currently, anything more interesting than plain Fact or Theory based tests might not work. Test discovery in the runner is handled by xunit2, hence the weasel word “might”. Fixing this is my next priority.

It currently only supports 8.2, as this is what I’ve been using to build and test. I’m not sure yet if I’ll add support for 8.1 back. Leave feedback on GitHub if you really want it – but leave a reason, too. After all, 8.2 is a free upgrade!

Annotations and live templates haven’t been updated yet, or even tested. They might work, they might not – so your methods might not be highlighted as in use, and the templates might not work (theory definitely won’t – it’s in a different namespace)

Moving to GitHub

I think it’s fair to say that xunitcontrib had a good run on CodePlex, but, well, GitHub is better. It’s a much nicer experience, everything is just easier, and well, GitHub won.

A very nice feature on GitHub is organisations. Brad and Jim have very generously invited the ReSharper runner to live under the xunit org. It doesn’t change the ownership, copyright or license of the project (xUnit.net is an Outercurve project, but the ReSharper runner is mine), it’s simply a nice place to live.

The name of the project has also changed. It always was more of a ReSharper project than a contrib project, so the name is now “resharper-xunit” to reflect this. I’ll still be using “xunitcontrib” as the name of the ReSharper packages, though.

This is a big update, and a long post (but there’s screenshots!) so let’s cut straight to the chase. If you want a download link, go straight to CodePlex and grab it. If you’re interested in what’s new, let’s start with a bullet list:

Where’s the ReSharper 8.0 EAP support? The code’s there, it’s ready to go. But the current EAP builds have an issue that prevents the test runner being loaded unless it’s in the product install directory, so it can’t work for plugins right now. This should be fixed soon.

Right. Screenshot time.

PropertyData attribute support

The PropertyDataAttribute tells xunit to get data for a Theory parameterised test by calling the named property’s getter. It relies on a magic string. And we all know that magic strings break when you rename the thing its referring to. Wouldn’t it be nice if ReSharper could help here?

And of course, it can. This release tries to add a reference between the string value of the attribute declaration and the property it’s referring to. So, if ReSharper can’t find a property with the same name as the string, it’s displayed as an error:

If it can find it, then it happily takes part in the ReSharper world of loveliness – ctrl-click navigation, included in find usages, renamed during refactoring, and it even marks the property as in use.

And there’s code completion (oh look! That’s Visual Basic!):

And of course, it works with the overload that takes a type name as well as a property name:

Traits (categories) support

NUnit and MSTest both have a Category attribute. This takes a single string value, and ReSharper can use this to group tests, so you could, for example, run all “logging” tests at once. The xunit approach is a bit different, though. Instead of a Category attribute, xunit has the more general Trait attribute, that takes two string parameters, a name and a value. It’s trivial to model category with this, just use a name of “category”, but what do you do with the other trait values?

This answer is of course obvious, especially in hindsight (this post on how the TFS team had implemented it in the Visual Studio test runner was the light bulb moment for me). If you have two strings, and need to make one, just concatenate them. So, name/value traits are formatted as “name[value]” and this is treated as the ReSharper category. The name “category” gets special treatment. As long as the name is “category”, regardless of case, only the value is used. This allows for usage such as [Trait(“category”, “logging”)] to be grouped under the “logging” category, and something like [Trait(“owner”, “matt”)] to be grouped under the “owner[matt]” category. Note that only “category” gets the special, case insensitive, treatment – “owner” is not the same as “Owner”.

And there is code completion here too, on both the name and value (yes, more VB!):

RunWith support

This has been one of the longest outstanding issues. The RunWith attribute allows you to change how the tests for a particular class are run – you get to take over the test discovery process. The default process looks for all methods that are decorated with a Fact or Fact derived attribute, but you could create a test class command that returns tests that being with “Test” or “Should”. Or one that calls a method in the class to get the tests to execute, or which runs the tests in a specific order.

The tricky thing here is that the ReSharper plugin can only go on what’s in the source code – it can’t run arbitrary code to find out what methods (if any!) should be tests. And it might be a random process, that changes each time the code is run. Sounds a lot like Theory support, in fact.

When the plugin encounters a class decorated with the RunWith attribute, it marks the class as being a test class, but it doesn’t mark any methods as test methods, even if they’re decorated with the Fact attribute – this could be a red herring if the custom RunWith test discovery is based on method name.

When the class is run, the methods are dynamically added to the test runner results window. Double clicking on the test will navigate to the appropriate method, and individual methods can be run and debugged just as with normal methods and theories. (The PrioritizedFixture shown here comes from the xunit samples)

Live Templates

There have been live templates in the release for years, but they’ve been a bit neglected – they were still in the old ReSharper 4.1 xml format! I’ve given them a bit of love and updated and added to them, and they’re now a first class part of the plugin.

The best bit is that you don’t have to do anything to get them – the plugin adds them to the settings system automatically. Just go to the Templates Explorer and see all the new templates tagged “xunit” in the C# section.

Wherever possible, the shortcuts have been chosen to be mnemonics of the assert, “ae” is Assert.Equal, “at” Assert.True, “athr” Assert.Throws, “adnt” Assert.DoesNotThrow, and so on. There are a few more for creating test methods, including “fact” to create a method decorated with [Fact], “fa” expands to [Fact] (Fact Attribute), xtc expands to an Xunit Test Class, and so on. Check them out in the Templates Explorer.

If you don’t like the shortcuts, or the code that’s generated, simply edit them. The changes are saved in your settings by default, and don’t override the shipped templates. (Although if you’re editing them, why not send me a pull request with the changes?)

But if you really don’t like the templates, you can disable them completely, by going to the ReSharper –> Manage Options dialog and unchecking the “xunitcontrib templates” settings layer. This removes them from the system completely.

External Annotations

External annotations have been in the zip file for years, too, also neglected. ReSharper’s analysis engine is very clever, but it can be even smarter if you give it more context by annotating your code with certain attributes. You can tell ReSharper when a return value or parameter will be null or not null, or if the method will throw an exception if a certain parameter is null, not null, true or false. You can even say that a lambda is being used immediately, rather than stashed away for later use, which limits the scope of a closure, and stop ReSharper showing you the dreaded “access to modified closure” or “implicitly captured closure” warnings.

And if you don’t have access to the source to add these attributes? Then you create an external annotations file, which maps annotation attributes to type members. The annotations for xunit have been updated to xUnit 1.9.1, and are now automatically installed.

Aborting test runs

This was something I hadn’t noticed about the test runner. When the tests are running, hit the stop button. The button changes to a hand icon, but the tests keep running. Hitting the button a second time stops the tests immediately. What’s going on here?

The first hit signals to the test runner process that you want to abort, and waits for the test runner to comply. If it doesn’t stop in good time, and you hit it again, it forcibly kills the test runner process.

Turns out, the xunit runner wasn’t paying any attention to the signal to stop, so the only way to abort a test run was to kill the process. And then the shadow copy cache didn’t get deleted, leaving folders lying around in your temp directory.

This release now plays nicely with the stop button, and will stop after the current test is run. If you forcibly end the run, the cache gets cleaned up correctly.

Update notifications

And finally, because no plugin should be without them, the plugin will now check for updates, and let you know if one is available.

What’s next?

I need to test these changes in the standalone dotCover test runners, and get a 1.0 release out there. And then, it’s full steam onto version 2.0, which will be all about the xunit 2.0 rewrite.

ReSharper 7.0 brought full support for Visual Studio 2012. One of my favourite little features was preview tab integration.

The preview tab is reusable, rather than the normal, persistent editor tabs. It displays one file at a time, and the content is replaced when a new file is navigated to. This is a great way to reduce the number of open tabs in the editor, especially if you’re just browsing a codebase. If you want to edit a file, just start typing, and it seamlessly becomes a normal editor tab.

Also, it’s a nice purple colour.

ReSharper 7.1 changed the behaviour, using the preview tab much less, and now normal editor tabs abound and multiply.

I’ve taken this rather personally. It’s messed up my workflow. I’m back to the bad old days of VS 2010, all too often declaring Document Bankruptcy and closing all windows, even those I’m editing on.

That’s not how I work. I read code as much as I write code, and tend to have to navigate to the right place before I can change anything. That can open more files than I actually need to work. The preview tab is brilliant for this – I get to read the file and navigate away without it cluttering up the editor. If I want to edit it, I simply start typing. And if I need to get back to a file that’s been replaced? Use ReSharper’s Recent Files popup, and bob’s yer uncle (you don’t use ctrl+enter, do you?)

Fortunately, as is the way with ReSharper, it’s extensible and can be (ahem) fixed. So, if you want to get back to purple preview tab perfection, here’s a nice little plugin that forces all navigation to go through the preview tab. Source and binaries are on GitHub, and you can find the download link on the Release Notes page.

(ReSharper 7.1’s use, or lack of use, of the preview tab isn’t arbitrary. Direct navigation opens a normal editor tab. Browsing from a tool window, by hitting enter, or ctrl+alt+up/down to browse previous/next occurrence uses the preview tab. I just happen to want direct navigation to use the preview tab, too)

I did, frankly. I passed a package folder path instead of a repository package path, and it just happened to work in NuGet 2.1. A change in the implementation details in NuGet 2.2 and it was now using the wrong path and couldn’t find the package to install. Garbage in, garbage out. This meant the reference wasn’t added, or a file reference was added and dependencies weren’t, usually silently, so you didn’t know something had gone wrong. The fix was quick, and I added better error handling and reporting, so if something goes wrong, it will now fail more visibly, and without falling back to adding file references.

The crashing VSTS DB Pro issue was much more interesting, because it’s seemingly unrelated to NuGet, ReSharper or the plugin. Three virtual machines later, I could recreate it, by using Windows 7 and not installing VS 2012. Or, more accurately, by ensuring that .net 4.5 wasn’t installed. I had compiled the plugin against .net 4.5, and when running with just .net 4.0 installed, it would cause VSTS for DB Pro to crash. I still have no idea why – there were no useful exception messages or any other clues. A quick recompile later, and no more crashes.

One little enhancement made it in – update checking (inspired by Julien’s ForTea T4 plugin). ReSharper will notify you when a new version is released.

So, apologies for the nasty bugs – get the latest version from the link on the Release Notes.

And it would be mean not to throw in a couple of bug fixes while I’m at it.

Firstly, there’s a fix to support filtering out test usages when using ReSharper’s Find Usages command. Frankly, I thought this was working ages ago, but it looks like I broke it some time back. Sorry folks.

When in the Find Results window, displaying the usages of a type, method, property, etc. you can filter out different types of usage – read/write, invocation, usage in attributes or documentation, and, what’s interesting here, usages in tests. This means you can hide any usage of a type from a test method or class. Simply toggle the “Show Unit Test Usages” menu item in the filter, and the Results window will hide and show the test usages (note that the test usages are using the test icon that appears next to the test method in the code editor).

The second bug fix relates to Theory data rows. xUnit.net uses class and method names as a means of identifying a test. This is guaranteed to be a unique value, as the test method doesn’t run more than once. For Theory parameterised tests, xunit uses the class and method name, but also uses a display name, which includes the parameter values, such as “ShouldBeUppercase(s: SAUSAGES)”.

There is a problem here. If you have two rows of data with the same value (“SAUSAGES”), the test runner treats these as the same test. This is a case of “doctor, doctor, it hurts when I do this”. Don’t do this. You’re passing in the same data, you’ll get the same results, it doesn’t matter if it looks like there’s only one test.

Or so I thought.

What happens if the data being passed in is an array of strings? xunit will call ToString on the parameter value, and come up with a display name like “ShouldBeUppercase(s: System.String[])”. Clearly, there are going to be collisions here. So now, the test runner keeps a track of names of tests it’s already run. If it tries to run the same test again, it changes the name, appending a call number, e.g. “ShouldBeUppercase(s: System.String[])” and “ShouldBeUppercase(s: System.String[]) [2]”, “… [3]” and so on.

And that’s it for this release. I smell a 1.0 version coming, finally. There are a couple of big ticket changes I want to make first. I want better support for PropertyDataAttribute (IntelliSense, find usages, ctrl-click navigation, etc) and InlineDataAttribute, and there’s a good memory optimisation I need to make for editing a large test file. If there’s anything else missing or broken, now’s a good time to let me know!

Exceptions were thrown when trying to parse a source file that contained static public properties and had a build action set to None. Rather a specific one, that.

The most interesting change is for placing tests in nested classes. When displaying a test method in the UI, it’s usually displayed as the class’s short name, e.g. For “Name.Space.ClassName”, you see “ClassName”. This scheme fails when you have nested classes, and multiple nested classes share a name, but have different parent classes. Like this:

making it very hard to distinguish which test was which. Now they display like this:

In other words, nested classes include the name of their parent class in the UI. I’m not exactly enamoured of the naming format; using a “+” to separate the class names isn’t very well known outside of Relection, but at least it gives you the chance to know which test is which. It might change in a future version - there’s an outstanding bug for ReSharper to improve the NUnit runner’s support for nested classes, and I’ll follow their lead.

Removed support for 5.1:

I started this project with ReSharper 4.1. I make that 7 releases (4.1, 4.5, 5.0, 5.1, 6.0 and the current 6.1.1 and 7.0). Going forward, I’m only going to support the last two major releases – i.e. 6.x and 7.x, so that means the current release has binaries for 6.1.1 and 7.0 RTM. You can still download the binaries to any of the old releases, and the source is still available in the repo, but they’re not going to see any new development.

Yes, the latest release of the xUnit.net ReSharper test runner provides much better Theory support (theories are xunit’s parameterised tests). Up until now, the support has been so poor (i.e. non-existent) that it’s not always been possible to tell if a theory has failed, and if it has, which one did and why. It’s been the biggest issue and most requested feature since the project began.

Well, that’s all in the past now. Here’s the latest version running some Theory tests:

Yep, theories are now displayed as child nodes under the parent method, as you’d expect.

It provides support for any kind of Theory – data from attributes, properties, spreadsheets, databases and any other kind of custom Theory or multiple row producing custom Fact-based attribute you might like to write. It also works with dotCover to provide code coverage. (I’ll update the standalone dotCover runners shortly)

This release works with 6.1 and the current build of 7.0 (that is build 82). The download does not include runners for 6.0, 5.1 or earlier versions of ReSharper. Those versions are packaged up in an archive release. The idea is to only support the latest versions of the last two major releases of ReSharper.

I've just checked, and I've been working on the xUnit.net ReSharper test runner for nearly 3 and a half years. I've always enjoyed writing tools and plugins, possibly more than server side or front end coding, the stuff that's paid the bills. I like building tools that help make people more efficient, and then gets out of the way. I love the idea of tweaking or extending a tool and making it to do new and interesting things.

One of my all time favourite tools is ReSharper. I’m a huge fan, use it constantly, and don’t know how I managed to develop without it. I know ALL the keyboard shortcuts and everything.

Which means that I'm VERY excited that from the start of July, I'm going to be part of JetBrains' .net developer evangelism team, working with Hadi, Jura and the others. Specifically, I'll be focussing on the extensibility features of ReSharper and the other .net tools (dotCover, dotTrace, dotPeek - yep, these guys are extensible too).

In other words, I'll be working on plugins.

Or rather, working with the whole plugin ecosystem. While it will involve writing, maintaining and publishing plugins, it's also a very community facing role. I'll be involved in, and provide support for, the existing community of plugin authors - helping out with issues and updates, gathering feedback, improving the ReSharper SDK. I’ll be encouraging new plugins by working on blog posts and tutorials, and participating in JetBrains events and community gatherings. Everything I can to improve ReSharper as a platform.

In some respects, this is quite a departure to what I've done in the past. I've worked in C++ and C#, a bit of Java and even some Flash (don't tell anyone). I've been a developer, team lead or architect in small code shops and several big enterprises. Now I'm going to be working from home (which beats the London commute I've been doing for the past few years), with a very social, community focussed role, working full time on the kinds of things I've been tinkering with in my spare time.

JetBrains recently released an EAP of dotCover 2.0. Up until now, dotCover required ReSharper to enable code coverage of unit tests – it integrated itself nicely in the ReSharper menus and UI, and would provide a coverage analysis of the code executed during a test run.

Starting with this 2.0 EAP, you no longer need ReSharper installed. DotCover provides support for its own test runner and test runner plugins. In fact, it comes bundled with a port of the unit test support and framwork from ReSharper, so you won’t actually be able to tell the difference.

If, however, you want to see what all the fuss is about, you can disable ReSharper’s unit test implementation, and use dotCover’s. Simply go to the ReSharper –> Options –> Unit Testing, uncheck “Enable Unit Testing”, and then go to the dotCover –> Options –> ReSharper Integration and uncheck “Hide all actions and tool windows related to unit tests in dotCover and use ReSharper’s implementation”. You will now be using the dotCover unit test implementation.

This version contains plugins for ReSharper 6.1, 6.0 and 5.1.3. I plan to only support the most recent revisions of the last two major versions, so that means 6.1 and 5.1.3. So, I’ll be moving 6.0 into the archive release very soon.

This is a small update. The important news is the support of ReSharper 6.1, but there was still chance to squeeze in a new feature.

ReSharper 6.0 added a nice little feature for running the derived instances of test methods defined in an abstract test method. To explain that a bit better, assume you’ve got an abstract base class that contains a test method:

ReSharper 6.x provides support for adding a marker next to AbstractBaseClass.TestMethod that gives a drop down menu allowing you to run TestMethod in one or all of the derived test classes. Something like this:

Unfortunately, 6.0 doesn’t provide enough support to get the naming correct everywhere in the UI. You can either specify the name as DerivedClass.TestMethod (which makes most sense when used in this drop down) or BaseClass.TestMethod (which reads better in unit test explorer and runner windows).

I’ve chosen to use BaseClass.TestMethod, because I think it’s more useful that the names are correct in the unit test explorer/runner windows (the built in nunit plugin uses DerivedClass.TestMethod). Annoyingly, this means the drop down isn’t terribly helpful: