Tag Archive: BizTalk Map Testing Framework

The out of the box unit test framework for BizTalk maps leaves a lot to be desired. Effectively a Visual Studio map unit test is the equivalent of choosing to validate your map, which involves running an instance of an input message through your map and validating the output of the map against it’s schema. If you want to check the values in the output message then it is up to you to implement your own logic for this, and if you want to test variations in some of the elements in your input message then you really need to be creating and managing more copies of the input message which can quickly escalate into a management nightmare. This doesn’t leave one with the utmost confidence in their map when making changes to it, nor does it assure one that all the functional requirements are being catered for.

The BizTalk Map Testing Framework provides a solution to many of these problems. It allows you to run a map against an instance of an input message and it will compare the actual output of the map against a pre-prepared expected output message. This is referred to as a base test.

It also allows you make multiple additional tests with variations of elements in the input message based on the same file, and validates the output against a variation of the output message also based on the same file. These variations are catered for through collections (one for the input message and one for the output message) of xpath statements which contain the elements which need to be varied for that given test case. The test implementor would then create test cases, each test case containing different values which will be replaced in the input and output message based on the xpath statements.

The BizTalk Map Testing Framework also comes with overridable methods called OnTestExecuting and OnTestExecuted which allow you to perform your own custom validation before or after the maps have been executed (note that OnTestExecuted might be run after the map has been executed but it will still be run before the test cases themselves have been executed). This is made easy as the framework also includes some handy helper classes that let you extract values from either the input or output messages to validate them, or even to replace the values in them with a pre-prepared value. This is especially handy when the output message will contain some dynamic values such as a GUID or the current date time and you want to perform some custom validation on them before replacing them with a static value so that the static comparison against the output message still succeeds.

Let’s illustrate this with an example. Our input message contains an element called EventType and if it has a value of U then a value of Update will be mapped to an element called EventType in the output message, if the value in the input is D the value in the output is Delete, and if the element has a blank value or doesn’t exist in the input then a value of Unknown should be mapped to the output, if the input has any other value then the output element shouldn’t exist at all. We also have a repeating element in the input message called EventCode which can exist upto 4 times, and an EventCodes element in the output message which is a concatenation of all the source elements but will be suppressed if there are no EventCode elements in the source message. We also want an element called Id in the output message to contain a valid randomly generated GUID.

The map and the example instances which we are going to use in our test class are as below.

Before we start we want to ensure that the projects containing our BizTalk maps and schemas are set to allow for unit testing. Do this by opening the project properties, browsing to the deployment tab and enabling unit testing. Ensure you deploy the solution after doing this.

We now need to create a test project, add a reference to our projects containing the BizTalk maps and schemas (only necessary if we want to do schema validation), add a reference to the MapTestFramework.Common dll, and references to the Microsoft.BizTalk.TestTools and Microsoft.XLANGS.BaseTypes dlls.

Next up we add a test class (remember to decorate your class with the TestClass attribute) and inherit from the MappingFixture base class contained within the MapTestFramework.Common namespace. You’ll now want to override the CreateMap method and the SourcePathBase and ExpectedPathBase properties as below so that the test class knows which map, input file, and output file it is going to be using.

Now we want to implement our base test which will perform a static comparison of our output file to the actual file that was generated. The base test should look like below.

When you attempt to execute the base test you will notice that it fails since the value of the GUID is randomly generated and thus doesn’t match the GUID in our expected output file. In order to cater for this we would need to override the OnTestExecuted method as below such that it validates the GUID in the actual output file (we’ll throw in schema validation here too just for fun, note that the below implementation of schema validation has many limitations but I have shown this for the sake of brevity), and then replace the GUID with the one that has been set in the static expected output file. Note the usage of methods within the XmlHelper namespace which are used to read in elements from the output file and to replace them as well.

The base test should now execute and return a succesful result.

Now we need to implement a test which checks for the different values in the EventType element. Create a new test method and instantiate an object of type MapTestCases passing two arrays of strings into the constructor, the first array containing a list of xpath statements that we want to use to manipulate the source message before the test is executed, and the second array containing a list of xpath statements that are used to manipulate the expected output file before comparing it to the actual output file. You can obtain these xpath statements from the Visual Studio schema editor or through a tool such as DanSharp XMLViewer. The MapTestCases object instantiation should look like the below (notice that I have removed all the namespaces in the xpath statements to make the validation namespace agnostic, it is your choice whether you want to do this or not).

You will now want to add test cases, which are different scenarios based on the xpath statements for the input file and the expected output file, as below.

In the above screenshot the first TestCase prescribes that if the EventType in the input message is U then the EventType in the output message must be Update. The third TestCase prescribes that if there is no EventType element in the input message then the EventType in the output message must have a value of Unknown. The fifth TestCase prescribes that if the value of EventType in the input message is OtherValue then there should be no EventType element in the output message.

Lastly you will want to instruct the test method that it should now execute the map test against all the test cases that you have added to the collection as below. Each one of the TestCases will be executed when the test method is executed and only if all pass will the test method be considered succesful.

The test for the EventCodes is somewhat different because there are repeating elements in the input message. The trick here is that if you want your xpath collection to include multiple iterations of the same node and you are expecting to set some of the nodes to null, then you must specify the nodes in backwards order. This is just logical because if you have 4 nodes and you set the first node to null then when you try to access the 4th node it will not exist because it is now actually the 3rd node. Thus you must start by setting the last node to have a null value and then work your way backwards. I have implemented this test method as below.

Some other things to consider.

I have not as yet been able to get the map test framework to work with a map in which the input message is an envelope schema containing an xsd:any node which links to an external xslt file. I have reverted to using Visual Studio map tests for this map.

Executing the map tests will result in lots of files being created in the expected output file folder. You will probably want to add a TestCleanUp method to your test class which deletes all the unnecessary files.

You will want to think of a smart folder pattern to store all your map files. This will make it a lot easier to manage all the input and output files.

You’ll want to ensure that your BizTalk projects have unit testing disabled in your release build which you use to create deployments for your QA/Production environments. The problem is that whenever you deploy based on your release profile and go back to your debug profile you will get a build error when you try to build your solution since the unit testable BizTalk artifacts aren’t deployed. I work around this by creating an extra build profile which has unit testing enabled on my BizTalk projects but doesn’t build my unit testing projects, and everytime I have deployed a release build and want to go back to debug I will first do a deploy on the intermediate build profile and then go back to the debug profile.

It is possible to have your map test case contain an xpath statement for a record/complex type element in your input or output message which contain multiple elements within it, and instead of just passing a string value into the individual test cases you can pass in the XML for all the contained elements (don’t include the XML tags for the record/complex type element itself).

I have recently been working on an agile BizTalk project which has required me to flesh out testing frameworks for all the different aspects of BizTalk. This has been a somewhat uncharted territory for myself in the past, though not for the lack of desire to explore this area, and was a great exercise for me. In this post I will aim to discuss some of the tools I have used, some of the reasoning behind why I chose to use those tools, and some notes about BizTalk testing in general. In later posts I will aim to do more of a deep dive into the tools mentioned.

The very first thing to mention is that in order to test your schemas, maps, and pipelines you will need to make a change to your project file. You will need to open the project properties, navigate to the deployment tab. On here, you will want to set the “enable unit testing” option to true.

What this does is change the code-behind for all your schemas, maps, and pipeline classes. By default all schemas inherit from the base class SchemaBase (if you want to prove this, just click the expand button on the left of any of your schema files in a BizTalk project in the Visual Studio solution explorer and view the .cs file), however when you change the enable unit testing property for the project, all the contained schemas will instead derive from the base class TestableSchemaBase. Before you start panicking about the fact that your schema has just been unnecessarily changed, the TestableSchemaBase class also derives from the SchemaBase class, it just implements a few extra properties/methods. Similar behavior can be observed for the code-behind for maps and pipeline artifacts as well. Using the Testable base class for your artifacts enables you to unit test them.

However, doing this also presents a problem. You’ll notice that once you enabled unit testing on a project that a reference to BizTalk.TestTools has been added to the project as well. On a typical non-dev BizTalk environment where you don’t have Visual Studio installed, this dll would not be in the GAC. This means that if you deploy your project with unit testing enabled to such a server, that you would encounter massive failures at runtime. One option is to of course make the missing dll’s available on those servers (you might find that you have good reasons to do this), but another somewhat cleaner approach is to only enable the unit testing property on your debug builds which will only be deployed to your development box, and ensure that it is turned off on the release builds which can then be deployed to your non-dev environments. Purists would argue that you are now releasing assemblies which are actually different from those that you unit tested, but the way I see it there has to be some degree of trust that we allocate to Microsoft to have gotten this right 🙂

On to the nitty gritty. The first type of test I tried to implement was unit testing for schemas. The idea of unit testing schemas is to validate instances of actual messages against a given schema, similar to right clicking on a schema file in the solution explorer and choosing validate instance. I tried to follow the textbook examples for the out of the box unit testing for schemas (see http://msdn.microsoft.com/en-us/library/dd224279(v=bts.10) for an example) however I very quickly ran into some major roadblocks.

When tests fail they don’t report back to you the reason for the failure. I consider this to be quite a hinderance as it then requires manual steps to be taken to find out the reason for failure.

If your schema imports types from another schema file then the test will always fail.

Clearly this is not quite acceptable so I started searching the internet for alternatives, and ended up implementing a static helper class that looks quite similar to those described here – http://stackoverflow.com/questions/751511/validating-an-xml-against-referenced-xsd-in-c-sharp. This allows me to very quickly whip up a test with just a few lines of code to assert whether a given xml file validates against a specified schema. Note that in the below example, you are able to specify whether the instance file is in native (flat file) or XML format.

For every variation in my input file that I wanted to test, I had to create and maintain an extra XML file. You’ll find your repository of XML files building up very quickly.

The out of the box implementation of map unit tests helps you to execute the map, allowing for validation of the input and the output files, but does very little in the way of allowing you to actually check the data in the output file. That is totally left upto you.

Once again, I decided to find out what those who came before me had decided to do, and it appeared that best of breed that suited my requirements was the BizTalk Map Testing Framework – http://mtf.codeplex.com/. I must say that I am a huge fan of this framework because of the great flexibility it allows. The way this framework works is that for each map you want to test, you supply a template input file and an output file. You then supply xpath statements for all the nodes in the input file you want to vary, and the all the nodes in the output file that would be changed as a result of said variations. You then create test scenarios, each one supplying values for the input nodes and expected values for the output nodes. This allows you to very quickly create a lot of tests for complex maps, and if you find that you are dealing with a very simple map which only requires a simple test then just skip the xpaths and make use of the auto generated base test which ensures that when the map is executed against the template input file, that the output file matches the template output file. The framework of course allows for actions to be taken before and after the map has actually been executed in order to cater for dynamic scenarios such as the output file containing the current time etc… and even provides some handy helper methods to read from and manipulate the output xml files before the test scenarios are executed. I will definitely post in more detail about this in the future.

I had a glance at the out of the box unit testing for pipelines, and it didn’t look too bad at all. However based on suggestions found around the internet I decided to make use of BizUnit 4.0 and the Winterdom pipeline testing framework which is very well documented here – http://blogdoc.biztalk247.com/article.aspx?page=a45b5fcd-a1fe-4219-844b-5e7e5660a4ec. Once more, I can’t express how much of a convert I am to this framework and how easy it is to generate tests with this. When setting up your test you can specify an input context file which contains all the context properties that should be applied to the message before the pipeline is executed (obviously only applies for send pipelines), the input file (or possibly files in the case of send pipelines which can be used for batching), and an instance config file which allows you to override the default parameters for the pipeline. Executing the test will result in the generation of the resulting output message and optionally a resulting context file which contains all the context properties against the message after the execution of the pipeline. Thanks to the robustness of the BizUnit 4.0 framework, it is very easy to implement validators against both of these files types.

Next up came integration testing. By its very nature BizTalk is very much a black box and most people think of it as a system which messages go into, and messages eventually come out of. In a large way this is what the focus of integration testing is, to evaluate the outcomes of a given scenario in a black box fashion. Once again I decided to use BizUnit 4.0 as it is a very robust and flexible framework, with a very full featured set of test steps to carry out common functions required in integration testing. I will have to write a whole other post regarding BizUnit 4.0 but I encourage all those interested in integration testing BizTalk or other types of integration projects to download and play around with the framework – http://bizunit.codeplex.com/

While we managed to create some pretty complex integration test scenarios, due to BizTalk’s black box nature we really had little idea whether we had actually managed to exercise all the different paths of our orchestration. In comes the BizTalk Orchestration Profiler tool (http://biztalkorcprofiler.codeplex.com/) which allows you to view which parts of your orchestrations have been exercised, and which areas you need to focus more testing on. A few tweaks are required to get the tool to work with BizTalk Server 2010, Colin Meade has made life easy for us by documenting the required steps on his blog – http://midheach.wordpress.com/2011/10/29/biztalk-orchestration-profiler-2010/.

Next on the list was performance testing. The options really boiled down to using the load testing framework that is part of Visual Studio Ultimate or using LoadGen 2007 (earlier versions were not an option for me as I was testing against WCF transports which are only implemented from the 2007 version onwards). In my case I had to go with LoadGen as the client did not have access to Visual Studio Ultimate, so that was an easy choice. Since I had already used BizUnit 4.0 to implement my integration tests, I decided that it would be nice to use BizUnit 4.0 as a harness for LoadGen as well. This also means that I could leverage of BizUnit test steps to actually validate the outputs of my performance test as well to ensure that all the records that I was expecting BizTalk to process were indeed being processed.

I hit a showstopper very quickly here. LoadGen 2007 was unfortunately released in a state whereby its containing dll’s are not signed with a strong named key and thus can’t be added to the GAC as required by BizUnit 4.0. This wasn’t an issue with earlier versions of LoadGen however as mentioned these weren’t an option to me as I needed to test against WCF transports. Back to google once again I was guided to the following article – http://geekswithblogs.net/jorgennilsson/archive/2009/01/03/loadgen-2007-with-bizunit.aspx. What this article describes is the process to disassemble the dll’s back to MSIL code, associate the IL files with a strong name key, and then update references in the dlls to also take note of the new strong name key. Doing this and adding the resultant dlls to the GAC means that BizUnit 4.0 and LoadGen 2007 now play very nicely with each other, and allows for complex performance testing scenarios to be catered for.

While LoadGen 2007 allows for the generation of load on a server and BizUnit 4.0 test steps can be used to validate the outputs, an important part of performance testing is to study the impact that the load has on the BizTalk and SQL environments. In order to do this I made use of the Performance Analysis tools in Windows Server 2008 and PAL (Performance Analysis of Logs – http://pal.codeplex.com/). The idea is to generate data collection sets from Performance Monitor while the server is processing the load and to then have PAL process the resulting files. It will then generate an HTML report for you which provides warnings when thresholds have been passed. While this helps you analyze and search for problems, there are many important counters which need to be spot checked manually rather than solely relying on the PAL output, which is merely a guide and a starting point (albeit a fantastic one).

A further requirement which was explored, however was since dropped was to look at using Specflow (which is a BDD or Behavioral Driven Development testing framework – http://www.specflow.org/specflownew/) as part of our BizTalk integration testing. Specflow allows for test analysts to specify tests scenarios using verbose statements which are based on the verbs Given (specifying prerequisites), When (specifying the actions to be taken in the test scenario), and Then (specifying the expected outcome of the test scenario). Each sentence in the test scenario will end up being bound to a specific method which will then implement the actual test logic. Each sentence in the test scenario can also specify variable inputs thus making the test methods quite reusable. I chose to use BizUnit 4.0 to actually implement my test methods for all the reasons I mentioned previously. Specflow also allows for your resulting trx test results file (assuming you are using MSTest to execute the tests) to be converted to a nice html format. There are of course many free tools on the internet that allow you to do similar things (and one of my colleagues has even written a BizTalk project to do exactly this, now why didn’t I think of that first :)), but it is a nice little extra feature that comes with the toolset.

While there are many benefits to such a testing approach, especially for agile projects, it should be noted that there would be some up front overheads, especially with the first set of test scenarios being catered to. My personal take on this is that the overhead mostly comes into play because each sentence on its own represents a test case, rather than the entire test scenario representing a test case. Thus you have to structure your test cases quite carefully, in a reusable fashion (the reusability which might not necessarily be realized in some cases) and you have to think how you are going to flow test context between your different sentences. The framework obviously caters for such things and is in no way limiting of what you are able to achieve, however it is up to each project to decide whether Specflow is a good fit for it or not. There is some fantastic documentation about Specflow and BizTalk here – http://social.technet.microsoft.com/wiki/contents/articles/12322.behaviour-driven-development-with-biztalk.aspx (and do be sure to check out the very thorough videos by Michael Stephenson).

One final thing which I never managed to play around with myself, however did have on my wish list to investigate was to have my integration tests snoop on ETW tracing. I have started using the CAT Instrumentation Framework in a large way (see http://blogs.msdn.com/b/appfabriccat/archive/2010/05/11/best-practices-for-instrumenting-high-performance-biztalk-solutions.aspx), especially in orchestrations to try to break down the black box barriers and get a peek into the internal state of the orchestration’s flow. The “Testing inside BizTalk by using ETW Tracing” project (http://btsloggingeventsinbi.codeplex.com/) leverages off the ETW tracing enabled by the CAT Instrumentation Framework, allowing for your BizUnit tests to be aware of what the orchestration are currently doing. This allows you to take testing in a whole new direction and tear down the black box walls.

Please let me know if you want to discuss any of these topics in further detail, I will definitely aim to post more on testing tools in the near future.