пятница, 23 января 2015 г.

Introduction

I am writing BenchmarkSuite the framework which helps to write benchmark tests in the same way like NUnit unit tests. You write a method which do the benchmark of you code, then mark it with [Bench] attribute and run the bench-console application. It search all the methods marked with [Bench] in the the assembly runs them several times, measures the metrics, calculates Mean, Standard Deviation and other statistical variables and outputs the results to the console and XML file. For testing the usage of the BenchmarkSuite library I've decided to benchmark various binary serializers for some common operations.

I've created SerializersBenchmarks project on github, got some well-known and some unknown binary serializers and start to write benchmarks for them. At the start that was funny, I've written a benchmark ran the console and immediately saw its results. But when number of binary serializers became more than 3, and number of test types grew it became a pain to add almost identical lines of code (which differentiate only by name of serializer and method to serialize/deserialize data) to every project. At this stage I decided to automatize the process and use the code generation

It creates serializer, then creates object of the serialized type and calls 10000 operation of serializing the type to a MemoryStream.

For every serializer and type it differs only few lines of code:

function name

creation of serializer

creation of type

calling method to serialize the type to a stream

To reuse common code I decided to write one base T4 precompiled template, derive from it in every benchmarking project and customize derived template for serializer needs. I used "Inheritance Pattern: Text in Base Body" from MSDN library to create base template.The code for template

What do this template do? For every type added in SerializedType array it creates text of the Serializing function, which serializes type to the memory stream. In the placeholders where Serializer should be created it calls the virtual method InstantiateSerializer(), which must be overrided in derived class and must write the code text which instantiates serializer. Then it calls virtual method Serialize() to fill the placeholder with Serialization code.

The next step was to create derived template, which would fill up all placeholders in base template. This was a little tricky. At the first, we must to say that template is ihnerited from base template with

<#@ template language="C#" inherits="BenchArrayBase" #>

Then we need to reference the base template assembly and base template namespaces in derived templates. To do it there are commands

Unfortunately, in most cases you don't know the absolute path to the referenced assembly, because you can place your project everywhere. Putting assemblies into GAC is not the good option to avoid this issue. But there is a solution. You can use project macros like ${ProjectDir} or ${TargetDir} in the assembly name and they will be evaluated into the absolute path. I added project 'SerializersBenchmarks' which contains base template as a reference to benchmarking projects and used

<#@assembly name="${TargetDir}/SerializersBenchmarks.dll" #>

as a reference in T4 template. If your base template located in the same assembly as the derived template you can use

<#@assembly name="${TargetPath}" #>

construction.

So I've added these lines at the top of the template

base.TransformText(); calls all transformations in the base template and outputs generated code. During the transformation it calls the overrided methods from the derived template, so I have to define these methods at the end of the file

As you guessed base.WriteLine() writes the line into the output which is generated by base template. That's all! Only few lines of code and we can add new serializer to our benchmarks without copy-pasting and replacing bunch of strings.

Issues with Monodevelop

When I start to use inherited T4 templates in Monodevelop I've found that Monodevelop did not support them, it threw an exceptions on trying to use derived templates. So I made a patch with fixes for Monodevelop, which was accepted in version 5.8. Also I made a patch which allow to regenerate all T4 templates in the project or solution (very useful when you made some changes in base template and need to update generated code in the whole solution). To do it right click on the Solution and Project and go to "Tools/Generate T4 Templates" option menu. If you read the article when monodevelop 5.8 is not released yet and want to use T4 inheritance, you can use latest dev monodevelop snapshot. To do it, add the Xamarin dev repository to you repos (see the instruction), and then do the following commands from the command line:

суббота, 3 января 2015 г.

In Visual Studio you can use macros like $(SolutionName) in *.csproj file. The full list of macros you can find in MSDN. Monodevelop also has got the macros, but the list differs from MSDN. Here is the list of the project macros I've found in Monodevelop sources. It's actual for Monodevelop 5.8, in future version the list can be changed.