v0.94.51023.0, 10/23/2010

This is the last release with Visual Studio 2008 and MSBuild 3.5 support for Pex and Moles. In future releases, we will only provide the Visual Studio 2010 integration support and MSBuild 4.0 for Moles. Pex and Moles will continue to support .NET runtimes 2.0, 3.5 and 4.0.

Hightlights

Pex Workflow Activity: integrate Pex into your TFS build easily by using the new Pex activity.

Various small bugs fixed.

v0.94.51006.1, 10/07/2010

Bug Fixes

MSBuild invocation of moles.exe exceed allowed command line length. We have implemented a custom MSBuild task that fixes the issue.

Moles create file names that are too long. Moles creates small project names in order to avoid hitting the MAXPATH issue.

v0.94.50921.0, 09/21/2010

Null Reference Exception when Assemblies could not be resolved. Moles would fail with a null reference when one of the dependent assembly was not resolved. Moles is still failing but with a nicer error message.

v0.94.50915.0, 09/15/2010

Highlights

MSBuild support for Moles. The generation of Moles assemblies is fully integrated into the build through MSBuild targets. No more checked in generated files! Read the upgrade instructions!

Improvements

Run Unit Tests along with Tests. Through the Pex Options dialog, you tell Pex to launch a unit test run in parralel with the Pex exploration. Best of both worlds in a single click.

Disable Automatically saving Tests. Another new option that allows you to run Pex without modyfing the generated tests on disk.Several bugs reported through the forums were fixed.

State Coverage. The state coverage metric, enabled by adding the [PexStateCoverage] attribute computes the ratio of field writes vs field read in any assertion. This coverage gives you an idea of the state that was created by never used in an assertion.

Upgrading to Moles v0.94.50915

This is a guide on upgrading to Pex/Moles v0.94 to previous version. The Moles build Visual Studio integration was completely changed in favor of using MSBuild.

Deprecated features

Moles does not support embedded sources only. All Moles files are always compiled into assemblies.

The attributes Disable, DisableGenerationOnSave, DisableGenerationOnBuild on the Moles element have been removed.

The 'Moles and Stubs for Testing' item has been remove. To add a moles file, go to the assembly references node, right click on the assembly to mole and click on 'Add Moles Assembly'. For mscorlib, right click on the 'References' node and click on 'Add Moles Assembly for mscorlib'.

Easy upgrade path

The easiest way to upgrade your solution is following:

Open your solution

Right click on the solution node and select Upgrade .Moles files...

Click Ok on the dialog and let Moles fix your projects for you.

Remove the Disable... attributes from each .moles file if necessary

Clean and Build the solution again.

Double check that all the generated Moles assemblies or designer files have been removed from the source control. Besides the .moles files, no other files should be added to the source control.

To be really sure, open a command prompt at the solution root and run the following command that deletes recursively all Moles assemblies.del /s /q *.Moles.dlldel /s /q *.Moles.xmldel /s /q *.Moles.pdb

Manual upgrade

Delete all Moles assemblies and generated files from your disk and your source control

Remove all references to Moles and Behaviors assemblies

For each .moles file in your solution,

Right click on Properties

Change Build Action to Moles

Clear the Custom Tool field

Fix the markup as follows by removing the Disable, DisableOnGeneration, DisableOnBuild attributes.

Clean and Build

New Moles assemblies are placed in a subfolder called *MolesAssemblies* in each project.

Voila!

v0.93.50813.0, 8/13/2010

Highlights

Coverage progress bar in the Pex UI. We’ve added a progress bar that displays how much code Pex is covering. It should give you a good sense of the progress Pex is making and also allows opening a dynamic coverage report with a single click once Pex is finished!

Assertion coverage. Pex now collects coverage information about assertions in your code (thanks to the work of our summer intern, Dries Vanoverberghe). If a parameterized unit test executes without a single assertion executed, Pex will flag it.

Private Helper Methods. We have added (limited) support for exploring private methods. (To be precise, support for private methods in visible types with visible parameters.) The target scenario is to enable testing of helper methods which often arise after refactoring.

Test Promotion. When you like a generated unit test, click on the 'Promote' button to turn it into a 'manually written unit test'. Pex will move it to the main test class and remove its attribute.

Behaviors with a single click. We have packaged the Behaviors assemblies for mscorlib, System, System.Web and Microsoft.SharePoint. You will get them automatically by using the 'Add Moles Assemblies' shortcut.

Improvements

Pex tries to use human friendly characters and integers. Pex now prefers human readable characters over strange Unicode characters, if both cover the same execution path. Similarly, Pex now prefers small integers that do not cause overflows over big integers that cause overflows, if both cover the same execution path.

Moles assembly caching. The common system libraries are pre-compiled and cached so that you don’t have to worry about them. Their respective behavior assemblies are also precompiled and installed.

Moling an uninstrumented method in a single click. When Pex reports an uninstrumented method, Pex now also offers a code update that creates the necessary Moles assemblies to isolate from that method.

Tracking PexMethod signature changes. When you add or remove a parameter to a PexMethod, Pex automatically detects it and asks you whether you would like the generated unit tests to be deleted.

Allowing ‘Xml documented’ exceptions. We’ve introduced the PexAllowedXmlDocumentedExceptionAttribute attribute that automatically filters an exception as ‘expected’ if it is documented in the xml docs.

Suppressing Object Creation messages. When an object creation issue is not relevant in your context, you can now silence Pex by an attribute, which Pex will insert automatically when you click on the new 'Ignore' button.

Support for xUnit.net 1.6. We’ve upgraded our support for the latest version of xUnit.net.

Documentation

Pex Reference Manual. We’ve started to consolidate various sources of information into this reference manual. It should continue to grow overtime.

Unit Testing Asp.NET applications with Pex and Moles. A step-by-step walkthrough for writing isolated unit tests for Asp.NET. This tutorial shows how you can use Moles to deal with HttpContext.Current, HttpCache, etc. We also added a number of Behavior types for System.Web.

Bug fixes

“Moles host type stops working when the ‘Host Settings’ are set to x64”. We’ve corrected the bug that prevented Moles to run if the host settings were set to x64.

Moles may fail to compile when a COM embedded type is not resolved. Now, we transitively remove any reference to that type in Moles.

Generated source files should use ANSI encoding.

Existing unit tests may confuse Pex constraint solver.

v0.92.50603.1, 06/07/2010

Highlights

Rex integrates Pex.Rex is a tool that explores regular expression. It's algorithms have been integrated into the whitebox exploration engine of Pex. As a result, Pex is much smarter about generating strings that match a Regex.IsMatch call.

Preparable Types. It is possible to register a lazy setup method for a type. This method has to be static and will be invoked by Pex before any method of the given type is executed. This is a great way to mole parts of your system on demand.

Silverlight and Silverlight for Windows Phone support (Alpha). Right-click on any Silverlight project and hit 'Run Pex'. Pex will run your Silverlight code on the desktop CLR and start generating tests for it. The unit tests can be executed using the Silverlight unit test framework from http://silverlight.codeplex.com.

Improvements

New SharePoint Unit Testing with Behaviors tutorial. A step-by-step tutorial on leveraging Behaviors to test SharePoint Foundation code. We've added a number of missing behaviors and improved the initial experience of behaviors. Give it a try and see how behaviors will change your way of doing unit testing.

New Pex and Code Contracts tutorial. A step-by-step tutorial on using Code Contracts and Pex to make your code more robust and have better documentation. The tutorial comes with the installer and is on our documentation page.

PexAssert.AreEqual(bool, bool). checks the value in a way such that Pex tries to achieve MC/DC coverage.

Moles for System.Configuration. Moles can deal with ConfigurationManager and ConfigurationSettings.

Static constructor and Finalizer support for Moles. It is possible to tell Moles to completely wipe out the implementation of the static constructor or the finalizer through Moles.

Moles bundles Microsoft.Pex.Framework.dll. It is possible to re-execute unit test generated by Pex by only installing Moles on the machine.

Bug Fixes

Moles: external alias generated by the Moles code generator should not be C# keywords.

v0.91.50418.0, 04/19/2010

Highlights

Visual Studio 2010 RTM Support! Pex and Moles happily work in the final version of Visual Studio 2010.

Alpha x64 version available! This is the first release where the x64 version of Pex and Moles is available. The installer marked as 'x64' will install both the x86 and x64 tools so that you can execute tests as native 64bit processes or 32bit processes in WOW64.

Improvements

Add Moles Assembly with a single click. Right click on any assembly reference and select 'Add Moles Assembly' to add the .moles file for that assembly in your project.

Support for ref arguments in Pex. Pex can explore methods that have 'ref' arguments in their signatures.

New filtering mechanism in .moles files. We realized that the way types could be filtered in the .moles files was really confusing. This version support a new Types node under StubGeneration or MoleGeneration that allows to clear, add or remove types in the pool of types.

Smarter Moles Assembly regeneration. The Moles Visual Studio AddIn uses various hashes to avoid re-generating the .Moles dlls when unnecessary. Overall, you should see a lesser impact of Moles in your development cycle.

Support for Test Impact and IntelliTrace. Data collection for Test Impact and/or IntelliTrace now works for tests executed under the Moles host type.

Better SharePoint Behaviors. We've added a number of missing types in the SharePoint behaved types.

Clashing Type Names: Moles can deal with type name sharing the same full name in multiple assemblyes.

Long Identifier Names: Moles can deal with long type names.

The Moles HostType threading apartment state should be STA by default.

Incorrect overload resolution: in some scenarios involving overloaded abstract methods, Stubs would not properly be generated.

v0.90.50303.0, 03/03/2010

First release as a Visual Studio 2010 Power Tools. Pex and Moles are now availabe under Power Tools licenses. Pex is available for MSDN suscribers, while Moles is available to all developers.

New documentation! All of our documentation was rewritten, validated and improved. We have step-by-step tutorials for Moles, Pex, reference manuals that are just waiting for you.

v0.24.50222.1, 02/22/2010

Improvements

The performance of the Moles Host for Visual Studio Unit Test 2010 has been dramatically improved.

When displaying an exception in the Pex exploration result view, give better hint about nested exceptions.

Bug fixes

The wizard no long produces invalid comments when long type names were involved.

v0.23.50215.1, 02/15/2010

Improvements

Visual Studio 2010 RC support

It is now possible to analyse results in the explorer tree view while Pex is running without the focus being stolen.

The naming scheme of generated tests has been slightly updated to make the generated tests more stable:

when a test throws an exception, ‘Throws’ + Exception type name is added to the generated name a hashcode of the input/output behavior is appended to the tes

Added PexChoose.Array<T>(int length) method to choose a non-null array of a particular length.

Added EventLog and Registry behaved types.

Bug fixes

Cannot collect code coverage with Moles under VS2010.

The save button does not get refreshed in the explorer tree view.

A deadlock may occur while in the logging infrastructure.

Fixed report folder recycling.

Fixed support for NUnit in Visual Studio

Breaking changes

We’ve moved MoleRuntime.CreateContext() out into a separate class MolesContext.Create(). MoleRuntime is a class that contains the methods to support the Moles infrastructure and should not be used directly by the user. If you are using Pex with Visual Studio Unit Test, this change will not impact you.

We’ve removed the pex.exe /runner option and replaced it with dedicated executable. You can use moles.runner.exe or moles.runner.x86.exe to execute unit tests using Moles for various unit test frameworks.

v0.22.50128.1, 01/28/2010

Improvements

Interaction with the Source Control provider has been significantly improved for Moles

Support for Moles of nested types.

Improved logging in the Host Type

Improved Behaved collections

Added Behaved types for mscorlib and System.Web types

Bug fixes

Updated several outdated section in the documentation

Updated missing classes in the SharePoint samples

Fixed a limitation of the profiler that would silently fail to instrument certain method

Breaking Changes

We have formalized the naming convention of Moles and Stubs and found some holes along the way. Some mole method might have a new name under this version of the compiler.

v0.21.50115.2, 01/15/2010

Bug Fixes

The Moles VsHost fails to execute unit tests in different assemblies.

Pex deletes the report folder it is currently writing to.

Support for named indexers in Stubs

Fixed bugs in how Pex reasons about System.Convert.ToBase64, DateTime

Incomplete support for protected members in the stubs generation

Breaking changes

The Stubs framework was renamed to Moles framework. We have decided to make the Moles the center of the framework and as a consequence, renamed ‘Stubs’ to ‘Moles’ and 'Beaver' to 'Behaved'. (This does not mean that we encourage writing untestable code, as Mole help to make it testable. You should still refactor your code to make it testable whenever possible, and only use Moles when that’s the only choice). See below for the list of steps to upgrade your applications.

BaseMembers in Moles have been deprecated: this helper is not useful as it can be achieved in a better way through a constructor. We decided to remove it to reduce code size. The second reason is that BaseMembers would only work for types inside of the same assembly, which might seem inconsistent.

PexGoal.Reached is replaced by PexAssert.ReachEventually(). The PexGoal class has been integrated into PexAssert through the ReachEventually method which should be used with the [PexAssertReachEventually] attribute.

PexChoose simplified: we’ve simplified the PexChoose API; you can now get auxiliary test inputs with a single method call: PexChoose.Value<T>(“foo”).

Migrating from previous version of Pex

Since we’ve renamed Stubs to Moles, any existing .stubx files will not work anymore. Take a deep breath, and apply the following steps to adapt your projects:

change the project reference from Microsoft.Stubs.Framework.dll to Microsoft.Moles.Framework.dll

In general, the ‘Fallback’ prefix has been dropped in the following methods:

rename FallbackAsNotImplemented() to BehaveAsNotImplemented()

rename class MoleFallbackBehavior to MoleBehaviors

rename class StubFallbackBehavior to BehavedBehavors

v0.20.41218.2, 12/18/2009

Beavers: With stubs and moles, we provided a framework to write record and replay tests against any .NET type, interface, static, sealed etc... Beavers take you to the next level and allow you to write simple implementations with behavior (hence the ‘B’eaver) that can be used to replace the external system during testing. The benefit of using Beavers is that the resulting test cases are much more robust to code changes since the tests specify the state of the system, rather than a sequence of method calls and outcomes. We will talk a lot about Beavers in the future so let’s take a look at an example. The following snippet uses Moles to redirect the constructor of a Bank class, then sets up the GetAccountById(int) method to return an instance of IAccount, etc…This is a typical test that is based record and replay. The problem with this technique is that it is really fragile with respect to code changes: if the code calls different methods, the test will break.

The same behavior can be specified using Beavers. Beaver are simple implementations that simulate that capture enough behavior of the original code for testing. Using Beavers, one can specify the state of the system. The functional behavior is re-factored away in a Beaver class and reused in all tests!

Global Event View: when executing multiple explorations, it was cumbersome to go through each exploration to investigate the events (instrumentation, object creation, etc…). We have added a view in the Pex Explorer tree view that centralizes the events for the entire run.

Beavers for SharePoint: The samples that come with Pex contains a number of Beavers for the SharePoint object model (SPList, SPListItem, SPField, etc…) that should help you get started with testing SharePoint with Beavers. As an example, you can compare a mole-based test (left) and a beaver-based test (right) for a simple SharePoint service. The snippets below simulate a site with one web, with one list, with one list item, with 2 fields with one value:

Moles extensions for NUnit, MbUnit and xUnit.net: we provide an extension for each unit test framework that allows to annotate a unit test that uses moles with a [Moled] attribute. This attribute takes care of reseting the mole context, etc... We also provide the source of those extensions in the sample, in case you are using another version of the unit test framework. The extensions are located in Microsot.Stubs.YYY.dll.

Bug fixes and Improvements:

The HostType crashes non-deterministically.

Visual Studio 2010 crashes when loading with a ‘DocumentSite already has a site’ error message.

The Stubs code generator uses a lot of memory and may crash Visual Studio. The Stubs generator now runs out of process and does not take down Visual Studio when it runs out of memory.

The Stubs code generator detects version changes of the Stubs framework to regenerate the stubs assemblies.

The Pex execution produces less warnings. We’ve reviewed our warnings and moved some, who did not matter, as messages.

When a Contract.Requires is failed multiple time in a stacktrace, do not allow it.

The object factory guesser has partial support for internal types and internal visible to (signed assemblies not supported).

The ‘add’ and ‘remove’ of events can be moled to deal with events.

Digger does not work in Visual Studio 2010 when no test project exists.

Breaking Changes: The Beavers introduced a number of refactoring that might impact users of Stubs and Moles. The .stubx file should regenerate themselves automatically, otherwise right click on the file and select ‘Run Custom Tool’.

Most types of the Stubs framework have been moved to sub namespaces.

IStubBehavior was renamed to IBehavior

StubFallbackBehavior was renamed to BehaviorFallbackBehavior

PexChooseAsStubFallbackBehaviorAttribute was renamed to PexChooseAsBehavedFallbackBehaviorAttribute

PexChooseStubBehavior was renamed to PexChooseBehavedBehavior

PexExpectedGoalsAttribute and PexGoal.Reached(...) were renamed and refactored into PexAssertReachEventuallyAttribute and PexAssert.ReachEventually(...);

v0.19.41110.1, 11/12/2009

Better Dev10 Beta 2 support! We’ve upgraded our support for Visual Studio 2010 Beta 2. This includes full support for .NET 4.0 Beta 2, the Visual Studio Unit Test HostType and the Visual Studio integration. Known Beta 2 issues: 1) When running Pex for the first time, the path to Visual Studio Unit Test is not found. You need to specify the Visual Studio ‘PublicAssemblies’ path (“c:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\PublicAssemblies”), 2) Don’t run Pex on v2 projects, Pex will crash.

Smooth generation of Stubs and Moles: when you create a stubs and moles assembly that is not a project reference, like one of the system assemblies, the stubs and moles code generator will automatically compile it into an assembly and add it to the project. This dramatically increases the performance of projects using Stubs and Moles.

Stubs and Moles for VisualBasic.NET: the Stubs and Moles may be used in VisualBasic.NET projects. In that case, they are embedded as a compiled assembly.

Moles support for NUnit, xUnit.Net and others: Unit tests involving Moles require the Pex profiler to work. We’ve added a new mode to our command line runner that allows to execute your favorite unit test framework runner under Pex. For example, the command “pex /runner=xunit.console.exe tests.dll” will launch the xunit runner for test.dll under Pex. x64 assemblies are still not supported.

Nicer Moles Syntax: we’ve added some little tweaks to the Moles API, such as an implicit conversion operator to the moled type, that simplifies the syntax. Here is a small example that showcases the current syntax: step 1 moles the Bank constructor, step 2 moles the GetAccount method and returns a mole of Account, step 3 moles the getter and setter of the Balance property.

Fluent Mole Binding: The Bind method returns the mole itself which allows it to be used in a fluent fashion. Interface binding is specially useful when you need to mole custom collections – just attach an array! For example, we attach an array of integers to a the MyList type that implements IEnumerable<int>.

Breaking Changes in Stubs and Moles:

The stubs and moles code generator tool is now ‘StubsGenerator’. You need to path this value in the property pages of your .stubx pages.

No more AsStub and AsMole: the AsStub and AsMole helpers were confusing and pretty useless so we decided to reduce the amount of generated code by removing them.

The naming convention for Stubs delegate fields always mangles the parameter types of the stubbed method, i.e. Foo(int) yields to a field called FooInt32. We used to be smart about adding parameters but we decided to make the scheme stable and symmetric with respect to the Moles.

The ‘FallbackBehavior’ property has been renamed to ‘InstanceFallbackBehavior’ in both Stubs and Moles to make it more explicit.

v0.18.41014.0, 10/14/2009

New Tutorials: Stubs and Moles, and Unit Testing SharePoint with Pex: If you felt that you didn’t know how to get started with Stubs, Moles, and Pex, then you will like the following two tutorials. They are bundled in the Pex installer, and you can also get them from our website:

Moles for Visual Studio 2010: The [HostType(Pex)] attribute, which is required to execute generated, or hand-written parameter-less test cases that Moles, now also works with unit tests in Visual Studio 2010.

Breaking Changes: The code generation of Moles has changed again. This might mean that you will have to recompile your solution, and adapt all existing uses of Moles.

v0.17.41006.7, 10/07/2009

Per-Instance Moles: Instance methods may be moled differently per object instance. This is particularly useful when multiple instances of same type need to behave differently in the same test. For example, let’s consider a simple class Bar that has a ‘Run’ method to check that the values are not the same. The important point here is that we would not be able to deal with such cases if we could not mole the behavior on a per-instance basis.In the test case of Run, we create to instance of MBar (the mole type of Bar) and assign two different delegates to the ‘Value’ getter. Moles are active as soon as they are attached (unlike the previous version) so we can just call the ‘Run’ method once the moles are set up. In the code below, we are using the C# Object Initializers feature that allows us to set properties of a newly created object within curly braces after the ‘new’ call.As you may have noticed, we just wrote a parameterized unit test that takes arbitrary ‘left’ and ‘right’ values. We then simply execute Pex to find the relevant values to cover the ‘Run’ method.

Moles for Constructors: In same cases, objects created inside of other methods need to be moled to test the actual program logic. With moles you can replace any constructor with your own delegate, which you can use to attach a mole to the newly created instance. Let’s see this with another simple example where a ‘Run’ method creates and uses an instance of a ‘Foo’ class.Let’s say we would like to mole the call to the ‘Value’ property to return any other value. To do so, we would need to attach a mole to a future instance of Foo, and then mole the Value property getter. This is exactly what is done in the method below.We then run Pex to find that 2 different values are needed to trigger all code paths.

Mole interface binding: All the member implementations of an interface may be moled at once using the new ‘Bind’ method. This smells like duck typing with type safety as the Bind methods are strongly typed. For example, we want to mole a collection type (Foes) to return a custom list of Foo elements (which need to be moled too). The goal is to test the sum method.In the parameterized unit test case, we create an instance of Vector, which implements IEnumerable<int>. Then, we can simply bind the ‘values’ array to the mole to make Vector iterate over the array. The call to Bind will mole all methods of MVector that are implemented by the ‘values’ parameters, effectively redirecting all the enumeration requests to the ‘values’ array.When Pex runs through the sample, it correctly understand the data flow of the parameters through the moles and finds inputs that break the ‘sum >= 0’ assertion.

Compilation of Stubs assembliesWhen one needs stubs or moles for a system assembly, it does not really make sense to re-generate the stubs each time the solution is loaded. To that end, Stubs ships with a command line tool that lets you easily compile a stubs assembly into a .dll: stubs.exe mscorlibOnce the stubs assembly is compiled, i.e. mscorlib.Stubs.dll is created, you simply have to add a reference to it in your test project to start using it. In future versions of Pex, we will provide a better experience to support this scenario.

The code generation of Moles was significantly changed. This might mean that you will have to recompile your solution, and adapt all existing uses of Moles.

v0.16.40915.5, 09/15/2009

Moles, a lightweight detour framework: Moles is a new extension of the Stubs framework: it lets you replace any .NET method (including static methods) with your own delegate. Because an example is worth more than a thousand words, here is snippet that shows how to make a static method (in this case, WCF OperationContext.Current) return a predefined value:Want read more about this sample? here is the full walkthrough.The moles framework is not yet feature complete; moles does not support constructors and external methods. Some types of mscorlib cannot be moled as they interact too deeply with the CLR.

Pex gets its own HostType: A HostType is a feature for the Visual Studio Unit Test framework that lets specific unit tests be run under specialized hosts such as Asp.Net or VisualStudio iself. In order to create reproducible test cases using Moles, we had to implement a HostType that lets tests be run under the Pex profiler. This is very exciting because it also opens the door for many uses of Pex such as fault injection, dynamic checking, etc… in future versions of Pex. When generating test cases with Pex, all the necessary annotations will be created automatically. To turn the Pex HostType on hand-written (non-parameterized) unit tests, simply add [HostType(“Pex”)] on your test case.This feature only works with Visual Studio Unit Test 2008.

Select your test framework: the first time you invoke ‘Pex’ on your code, Pex pops up a dialog to select your test framework of choice. You can select which test framework should be used by default and, more importantly, where it can be found on disk. If you do not use any test framework, Pex ships with what we call the “Direct test framework”: in this “framework”, all methods are considered as unit tests without any annotation or dependencies.These settings are stored in the registry and Pex should not bug you again. If you want to clear these settings, go to ‘Tools –> Options –> Pex –> General’ and clear the TestFramework and TestFrameworkDirectory fields:

Thumbs up and down: We’ve added thumbs up and down buttons in the Pex result view. We are always looking for feedback on Pex, so don’t hesitate to click them when Pex deserves a thumbs up or a thumbs down.

Performance: Many other performance improvements under the hood which should avoid Pex hog too much memory in long running scenarios.

Miscellanous improvements and bug fixes:

Support for String.GetHashCode(): we now faithfully encode the hashing function of strings, which means Pex can deal with dictionaries where the key is a string.

Fewer “Object Creation” messages that are not actually relevant.

In VS 2010, the package used to trigger an exception when the solution loaded. This issue is now fixed.

In Visual Studio when an assembly had to be resolved manually (i.e. that little dialog asking you where an assembly is), Pex remembers that choice and does not bug you anymore.

v0.15.40714.1, 07/15/2009

Better memory management and bug fixes. A lot of issues reported in the MSDN forums were fixed.

Breaking changes: Microsoft.Pex.Stubs.dll does not exist anymore and has been integrated into Microsoft.Pex.Framework.dll, PexExplorableFromFactoriesAttribute renamed to PexExplorableFromFactoriesFromAssemblyAttribute

v0.14.40610.2, 06/11/2009

Fix Suggestions for Invariant Methods: When you use Code Contracts, then Pex can now help you to add missing invariants, just as Pex already inferred missing preconditions in the past. Take for example the ArrayList class that comes with the samples of our recently updated tutorial slide deck. This class has the following fields:When you add an empty invariant method,and enable Runtime Checking of contracts, then Pex will create instances of this class using reflection while making sure that the invariants holds. However, with an empty invariant method, Pex will be able to create illegal states, e.g. where the _items array is null, which may cause a NullReferenceException, and other issues. To get the following test cases, we let Pex explore the Add method of our ArrayList class:When you look at the details of the failing test cases, then you will see that Pex now offers the option to add invariants directly into your [ContractInvariantMethod]:

Natural Properties for Stubs: The logic to handle properties with setters and getters generated by Stubs has been slightly improved so that it is possible to force properties to have as if they had a backing field.When a getter or a setter of a property is invoked, Stubs checks whether the getter delegate or the setter delegates have been set. If not, i.e. no custom user behavior, it queries the fallback behavior for an initial value of the property. If the fallback behavior successfully returns a value, Stubs generates a closure and binds the getter and setter.For example, here’s a simple IFoo interface with a Bar property:One could write the following parameterized unit test:Since we expect the Bar property to behave as it was wrapping a field, we don’t expect the assertion to be raised. When we execute this test with Pex, and the Pex fallback behavior, Pex will choose a value for ‘Bar’, and this value will be ‘stored’ the property. Therefore, we get as expected 2 test cases:

Non default constructor support for Stubs: In previous versions, Stubs could not generate stubs for classes with no default constructor. This is no longer the case, Stubs will generate one public constructor for each public or protected base constructor.

Visual Studio addin does not find attributes when they are globally qualifie

Added missing substitutions for System.Threading.Interlocked member

Stubs automatically imports missing reference

When an object has an invariant method (from Code Contracts), Pex would emit assertions using the private field

Editorial issues to patterns paper

Contracts no longer kill the process when the runtime rewriter was not executed (.net 4.0

Fixed bug in generic type guesser

Fixed samples for the latest version of Stubs

v0.13.40528.2, 05/28/2009

Seeding the Exploration Part 1: Unit Tests as Inputs

Before using heavy-weight constraint solving to explore hard-to-reach execution paths, Pex can now leverage already existing unit tests that call a parameterized unit tests: Pex scans their body to extract the parameter values, and then Pex uses these values to seed the exploration. (In the past, Pex would have seeded the exploration by simply using the default values for all parameters, and nothing else.)

Here is an example, where Pex starts the exploration of the parameterized unit test Multiply by reusing the values mentioned in the unit test MultiplySeed, as you can see in the first row of the table of generated tests.

As a (intended) side effect, Pex also reuses the tests that Pex might have generated during earlier explorations. In effect, Pex never has to start from scratch again, but instead can reuse knowledge from earlier explorations.

There is a big limitation in this first version of the new feature: Pex can only use values of primitive types, and arrays of primitive types as seeds, and the values must be passed as parameters, not via PexChoose. The unit tests must be in the same class as the parameterized unit test, marked as a unit test, and have a simple structure, i.e., they must just create the input data, and the first method call must be to a parameterized unit test.

Seeding the Exploration Part 2: [PexArgument(…)] to Fuzz Inputs and Help Pex

Pex can not only extract values from existing unit tests, but there is in fact a general API through which seed values can be given. In particular, you can use the new PexArgumentAttribute (and a bunch of related new attributes) to provide seed values.

Here is an example:

During the subsequent exploration, Pex will often reuse parts of the seeded values, which may make constraint solving easier for Pex. Also, when Pex cannot solve all constraints to leap beyond a difficult branch, then you can use this feature to help Pex by providing the values yourself.

Object Creation with Invariants

Code Contracts enable the specification of class invariants by writing an “invariant method” decorated with the ContractInvariantMethod attribute. For example, for a typical ArrayList implementation, this invariant method might read as follows:

public class ArrayList {

private Object[] _items;

private int _size; ...

[ContractInvariantMethod] // attribute comes with Code Contracts

protected void Invariant() {

Contract.Invariant(this._items != null);

Contract.Invariant(this._size >= 0);

Contract.Invariant(this._items.Length >= this._size); }

If you have such an invariant, and if you correctly configure runtime checking of contracts, then Pex will create instances of this type by first creating a raw instance, setting all private fields directly, and then making sure that the invariant holds. A generated test case for the Add method of the ArrayList class might then read as follows:

[TestMethod]

public void Add01() {

object[] os = new object[0];

// create raw instance

ArrayList arrayList = PexInvariant.CreateInstance<ArrayList>();

// set private field via reflection

PexInvariant.SetField<object[]>(arrayList, "_items", os);

PexInvariant.SetField<int>(arrayList, "_size", 0);

// invoke invariant method via reflection

PexInvariant.CheckInvariant(arrayList);

// call to method-under-test

arrayList.Add(null);

}

A limitation of the current implementation is that the type itself must be public, and the types of all fields must be public (but not the fields themselves.)

Surviving unhandled exceptions in Finalizers

Finalizers are especially problematic for test runners: there is no guarantee from the runtime when they will be executed, and when they throw an exception, the process terminates. This is not an uncommon scenario when you apply Pex to (buggy) code with Finalizers: It seems as if Pex suddenly crashes.

In this version, we have improved the robustness of Pex with respect to those failures. Pex effectively rewrites all finalizers of instrumented types with a catch handler that swallows all exceptions and sends it to Pex’ logging infrastructure.

v0.12.40430.3, 05/01/2009

Exploration Tree View: The exploration tree view displays the list of explorations to be executed, running and finished. It serves as a progress indicator but also as a smooth result explorer. When browsing through the tree, Pex will synchronize the exploration result view and the tree view.The tree view populates each namespace with the fixture types and exploration methods, and provide a visual feedback on the progress of Pex.When you browse through the exploration and generated test nodes, Pex automatically synchronizes the exploration result display. This makes it really easy to start from an high-level view of the failures and drill into a particular generated test, with stack trace and parameter table.

Partial Stubs: Stubs lets you call the base class implementation of methods as a fallback behavior. This functionality is commonly referred as Partial Mocks or Partial Stubs and is useful to test abstract classes in isolation. Stubs inheriting from classes have a “CallBase” property that can turn this mode on and off.Let see this with the RhinoMocks example on partial mocks. Given an abstract ProcessorBase class,we write a test for the Inc method. To do so, we provide a stub implementation of Add that simply increment a counter.

Miscellaneous:

PexAssume/PexAssert.EnumIsDefine checks that a particular value is defined in an enum.

Delegate As Parameters: Pex now understands Parameterized Unit Tests that take delegates as parameters. Pex will automatically generate a delegate instance and its behavior: if the delegate returns a value, Pex will generate a new symbolic value for it, track its use, and then generate different values depending on the program behavior. Let’s consider a simple method to illustrate this new feature. I wrote the following Parameterized Test method which takes a Func<int> delegate and throws an exception if the delegate returns a particular value: When executing Pex on Test, Pex will generate a Func<int> which ‘asks’ Pex to choose the returned int (it uses PexChoose under the hood). Therefore, for each call to that delegate, Pex has the liberty to return a diferent value. Based on how it is used, Pex will generate different values to increase coverage. In this case, Pex ‘chooses’ to return 123 on the first call, which is exactly what we need to cover the exception branch. The following code is generated by Pex; it starts by setting up the desired values, and then it calls the parameterized unit test.

Pex Explorer: Exception Tree View: We have started to work on improving the experience when applying Pex to a large number of explorations. To this end, a new window called Pex Explorer will show various views over the events produced by Pex. The Exception Tree View provides the tree of exception types that Pex found. This is really helpful to quickly drill through the (really) bad exceptions first.

Pex Explorer: Contract Failures Tree View: If you are using Code Contracts, Pex also provides a specialized view to sort the contract failures kind.

Events in Stubs:Stubs now support events:the stub simply expose the backing delegate fields (which hold the event delegate)as a public fields. As a result, one can clear, set, and invoke the event as any other member. Let’s see this with an example:In order to raise the SomeEvent in a test, we would simply have to invoke the backing delegate field, which happens to have the same name as the event: It’s that simple.

Recursive Stubs: Another common feature of mock/stub frameworks is to support nested invocation of members. Stubs now lets yourecursively invoke property getters. Instead of assigning the property getter delegate, you can usenew helper methods ending in ‘AsStub’ thattake care of allocating thenested stub, assigning it to the property getter and returning it. Let’s see this with an example: assume we want to test the little snippet below.In order to set up our stub, we would need to have IParent.Child return a stub of IChild, then set up IChild.Name to return ‘joe’. For property getters, Stubs generates a helper method that does just that:The ChildGetAsStub call instantiated a SIChild instance, assigned to the Child property and returned it. It’s that simple.

Fixed Bugs

.Stubx fail to generate the stubs. We fixed a bug with the registration of Stubs in Visual Studio.

Breaking Changes

Microsoft.Stubs.dll has been renamed to Microsoft.Stubs.Framework.dll. The public Stubs type has been moved under the Microsoft.Stubs.Framework namespace.

The generated stub class names are built by prepending ‘S’ to the type name (‘IFoo’ –> ‘SIFoo’). In the previous version, the ‘I’ of interface name was trimmed which sometimes created name clashes.

NUnit, MbUnit and xUnit.net supported out of the box: Pex now supports MSTest, NUnit, MbUnit and xUnit.net out of the box. Pex will automatically detect which framework you are using by inspecting the assembly reference list, and automatically save the generated tests decorated with the correct attributes for that framework.The default test framework can also be specified through the global options (Tools –> Options –> Pex –> enter the test framework name in TestFramework).

Writing Parameterized Unit Tests in VisualBasic.NET: While the Pex white box analysis engine works at the MSIL level, Pex only emits C# code for now. In previous releases, this limitation made it impossible to use Pex parameterized unit tests from non-C# code. In this release, we have worked around this problem by automatically saving the generated tests in a ‘satellite’ C# project. Let’s see this with an example. The screenshot below shows a single VisualBasic.NET test project with a Pex parameterized unit test:We can right-click in the HelloTest.Hello method and select “Run Pex Explorations”:At this point, Pex will start exploring the test in the background as usual. This is where the new support comes in: When a generated test comes back to Visual Studio, Pex will save it in a separate C# project automatically (after asking you where to drop the new project):The generated tests are now ready to be run just as any other unit tests!

Writing Parameterized Unit Tests from F#: Similarly to VisualBasic.NET, we’ve made improvements in our infrastructure to enable writing parameterized unit tests in F#. Let’s see this with a familiar example. We have a single F# library that has xUnit.net unit tests and reference Microsoft.Pex.Framework (project Library2 below). In that project, we add a parameterized unit test (hello_test):We can right-click on the test method name and Pex will start the exploration of that test in the background. Because of the limitations of the F# project system, you absolutely need to right-click on the method name in F# if you want contextual test selection to work. Because the project is already referencing xunit.dll, Pex will also automatically detect that you are using xUnit.net and use that framework. When the first test case comes back to VisualStudio, Pex saves it in a separate C# project:The tests are saved in the generated test project and ready to be run by your favorite test runner!

PexObserve: Observing values, Asserting values: We’ve completely re-factored the way values can be logged on the table or saved as assertions in the generated tests. The following example shows various ways to log and assert values:In the Observe method, we use the return value and out parameter output to automatically log and assert those values. Additionally, we add “view input” on the fly to the parameter table through the ValueForViewing method, and we add “check input” to be asserted through the ValueAtEndOfTest method. After running Pex, we get the following results:As expected, input, ‘view input’, output and result show up in the parameter table. In the generated test, we see assertions for the return value, out parameters and other values passed through the ValueAtEndOfTest method.

Code Contracts : Reproducible generated tests: When Pex generates a unit test that relied on a runtime contract, Pex also adds a check to the unit test which validates that the contracts have been injected into the code by the contracts rewriter. If the code is not rewritten when re-executing the unit test, it is marked as inconclusive. You will appreciate this behavior when you run your unit tests both in Release and in Debug builds, which usually differ in how contracts get injected.

Code Contracts: Automatic filtering of the contract violations: When Pex generates a test that violates a Code Contract pre-condition (i.e. Contract.Requires), there are basically two scenarios: the precondition was on top of the stack and should be considered as an expected exception; or it is a nested exception and should be considered as a bug. Pex provides a default exception filtering that implements this behavior.

Stubs: simplified syntax: We’ve considerably simplified the syntax of stubs by removing the ‘this’ parameter from the stub delegate definition. Let’s illustrate this with a test that stubs the ‘ReadAllText’ method of a fictitious ‘IFileSystem’ interface.

Stubs: generic methods: The Stubs framework now supports stubbing generic methods by providing particular instantiations of that method. In the following example, the generic Bar<T> method is stubbed for the particular Bar<int> instantiation:

Stubs and Pex: Pex will choose the stubs behavior by default: We provide a new custom attribute, PexChooseAsStubFallbackBehaviorAttribute, that hooks Pex choices to the Stub fallback behavior. To illustrate what this means, let’s modify slightly the example above by removing the stub of ReadAllText:If this test was to be run without the PexChooseAsStubFallbackBehavior attribute, it would throw a StubNotImplementedException. However, with the PexChooseAsStubFallbackBehavior attribute, the fallback behavior calls into PexChoose to ask Pex for a new string. In this example in particular, on each call to ReadAllText, Pex will generate a new string for the result. You can see this string as a new parameter to the parameterized unit test. Therefore, when we run this test under Pex, we see different behavior happening, including the “hello world” file:Note that all the necessary attributes are added at the assembly level by the Pex Wizard.

Miscellanous bug fixes and improvements

[fixed] Dialogs do not render correctly under high DPI

When a generic parameterized unit tests does not have any generic argument instantiations, Pex makes a guess for you.

When a test parameter is an interface or an abstract class, Pex now searches the known assemblies for implementations and concrete classes. In particular, that means that Pex will often automatically use the automatically generated Stubs implementations for interfaces or abstract classes.

Better solving of decimal and floating point constraints. We will report on the details later.

Breaking Changes:

The PexFactoryClassAttribute is no longer needed and has been removed. Now, Pex will pick up object factory methods marked with the PexFactoryMethodAttribute from any static class in the test project containing the parameterized unit tests. If the generated tests are stored in a separate project, that project is not searched.

The PexStore API has been renamed to PexObserve.

Pex supports versions of Code Contracts higher (strictly) than v1.1.20309.13, (which is the currently available version of Code Contracts!).

v0.9.40114.0, 01/16/2009 (Academic only)

Fixing bug in the academic installer that prevented to install on a machine without Visual Studio.

Stubs: support generated stubs for non-sealed classes and virtual methods.

v0.9.40105.0, 01/13/2009

Pre-Release License for Visual Studio 2008 Team Dev or Team Test: We have heard your feedback that it was hard to evaluate Pex on the Visual Studio 2010 Virtual Machine. As a result, the Pex research team and the Visual Studio Team Test team have decided to extend the Pex Pre-Release license to Visual Studio 2008 Team Dev or Team Test. Unlike the academic license, the pre-release license does not explicitly prohibit commercial use. Please note that this is a pre-release software with no support guarantees, but the Pex team does its best.

Test Framework Integration: Pex already had support for custom test frameworks in the previous version, but it only worked from the command line. We have added support for other test frameworks in Visual Studio as well and you will be able to generate tests for your favorite unit test framework. Similarly to the ASP.NET MVC Test Integration, any test framework can register itself through some registry keys (see extensibility guide (pdf)).

Partial Support for Visual Basic.NET and F#: Pex can analyze any MSIL code running on .Net, but it can only emit C# test cases for now. In previous versions, we restricted Pex in Visual Studio to C# projects because the Pex research team does not have the resources to implement and test the infrastructure for all other languages. We have heard your feedback: you can now generate parameterized unit test stubs from the project node of VisualBasic.NET and F# projects.

VisualBasic.NET also gets code digger, so that you can right-click in VisualBasic.NET files and launch Pex. However, when saving the generated tests, Pex will still generate a C# test project.

Better Regression Tests: When Pex tests a method that returns values, it automatically walks the object property graph (recursively) and stores the values as assertions in the code. If you make a breaking code change that can be observed through public APIs, the generated test suite is likely to flag it.

StyleCop, FxCop compliance: We have heard your feedback and made improvements to the generated test code so that it is clean with respect to FxCop and StyleCop.

Better support for instantiating value types: If a value type can be freely configured through the constructor and public properties, Pex will now automatically do that for you. Thank you for your feedback.

Other Features:

Code generation for ‘decimal’ type.

Ability to solve some constraints over decimal and floating point values.

v0.8.31021.3, 10/21/2008

New Experiences and Major FeaturesCode Digger - Code Understanding and Automatic Testing At Your Fingertips; read the announcement and the full Tutorial (PDF)Note: The Code Digger experience requires Visual Studio 2008 Professional, or better, with support for the MSTest unit test framework. Otherwise, you will see an "unexpected tool error" in this release. Stubs - A Simple Framework For .NET Test Stubs; read the announcement, and the full manual (PDF)

Other New FeaturesNew button with "clock" icon in main tool strip to make it easy to increase exceeded bounds. New "Delete Generated Unit Tests in File" menu item, available in files containing generated unit tests Console output of test cases is now shown in separate pane.

Bug Fixes, Performance Improvements, Minor Changes, Other MaintenanceSupport for forthcoming .NET FX 4.0Suppression of warnings which are hard to understand (they are still shown in diagnostics mode) Up to twice as fast Pex experience in Visual Studio on multicore machines More robust interaction of the VS AddIn with the Pex exploration process Lookup of substitution assemblies is now aware of different assembly versions Double click on row in test table selects top-level frame of exception stack trace instead of going to test code Fixed memory leaks in constraint solver (Z3) Bug fixes around generic types Many other bug fixes.

Known IssuesThe Microsoft.ExtendedReflection assembly is not listed in the Visual Studio Add Reference... dialog. If you need to reference it, click on Browse and locate the .dll manually. When using Pex with VS2010 CTP, note that the Pex samples project is still a VS2008 project. You should explicitly open it from VS2010, instead of double-clicking the solution file in the Windows Explorer.

FatalExecutionEngineError (-2146233082), or StackOverflow (-2147023895)This error might occur shortly after you invoke Pex, either from the command-line or Visual Studio. This is a known bug, and caused by a race condition in Pex. Usually, when you simply invoke Pex again, it will work just fine.

WPF issue: API restriction: '...WindowsBase.dll' has already loaded from a different location Pex may break your WPF applications. If you experience this issue, please uninstall Pex from the machine. We are working towards a fix for this issue.

v0.5.30611.1, 06/17/2008

First update of Pex!

Fixed bugs and other changes- Pex now complains gracefully when other profilers are active, e.g. certain mock frameworks. - The dynamic coverage view is no longer shown by default in Visual Studio, but it must be enabled (if desired) in the 'Tools' / 'Options' menu.

Visual Studio 2005 usersVisual Studio 2008 Standard (or higher) is required to run the samples and use the Visual Studio Add-in. If you have Visual Studio 2005 or Visual Studio 2008 *Express*, you can still use Pex from the command line. See the tutorial or the command line reference.

FatalExecutionEngineError (-2146233082), or StackOverflow (-2147023895)This error might occur shortly after you invoke Pex, either from the command-line or Visual Studio. This is a known bug, and caused by a race condition in Pex. Usually, when you simply invoke Pex again, it will work just fine.

Pex Wizard in Visual Studio may be slowWhen using the Pex Wizard in Visual Studio, devenv.exe might fully utilize a CPU for a long time. We will work on a fix for this issue in the next release. A workaround is to use the wizard from the command line and using the /testp flag.

v0.5.30516.0, 05/21/2008

First release of Pex!

Visual Studio 2005 usersVisual Studio 2008 Standard (or higher) is required to run the samples and use the Visual Studio Add-in. If you have Visual Studio 2005 or Visual Studio 2008 *Express*, you can still use Pex from the command line. See the tutorial or the command line reference.

FatalExecutionEngineError (-2146233082), or StackOverflow (-2147023895)This error might occur shortly after you invoke Pex, either from the command-line or Visual Studio. This is a known bug, and caused by a race condition in Pex. Usually, when you simply invoke Pex again, it will work just fine.

ClrMonitorFail (-667) This error might be result of an interaction of Pex with another installed Visual Add-in that also acts as a profiler, e.g. certain mock frameworks. This issue will be fixed in the next release. As a workaround, try to uninstall other Add-ins.

When the Pex Wizard generates parameterized unit test stubs, it inserts by default a call to Assert.Inconclusive(). This method throws a special exception that tells Pex (and MSTest) that the test is inconclusive. This mechanism forces the tester to review and validate each generated parameterized unit test by removing the Assert.Inconclusive() call. To suppress generating Assert.Inconclusive() calls in Visual Studio, go to the 'Tools' / 'Options' menu. Then select the 'Pex' / 'Wizard' pane and disable 'Make generated test inconclusive by default'.

Pex Wizard in Visual Studio may be slowWhen using the Pex Wizard in Visual Studio, devenv.exe might fully utilize a CPU for a long time. We will work on a fix for this issue in the next release. A workaround is to use the wizard from the command line and using the /testp flag.