Using Custom Performance Counters with Windows Azure Services

Windows makes available a wide variety of performance counters which of course are available to your Azure roles and can be accessed using the Azure Diagnostics APIs as I described in my recent MSDN article on Windows Azure Diagnostics.

However, it can be useful to create custom performance counters for things specific to your role workload. For instance, you might count the number of images or orders processed, or total types of different data entities stored in blob storage, etc.

I wasn’t actually sure that I would have permissions to create custom performance counters and it turns out that right now you do not – all the code described below works in Development fabric (where you are running with rights not available to you in the Cloud). Note as well that the symptoms when deploying to the Cloud with this code in place is that the role will continually be “Initializing” – it will never start.

Custom Performance Counters aren’t yet supported in the Azure Cloud fabric due to restricted permissions on Cloud-based Azure roles (see this thread for details) so this post describes how it will work when it is supported.

Create the Custom Performance Counters

You create your counters within a custom category. You cannot add counters to a category, so you have to create a new performance counter category using PerformanceCounterCategory.Create in System.Diagnostics. The code below shows how to create a few different types of counters. Note that there are different types of counters: you can count occurrences of something (PerformanceCounterType.NumberOfItems32); you can count the rate of something occurring(PerformanceCounterType.RateOfCountsPerSecond32); or you can count the average time to perform an operation(PerformanceCounterType.AverageTimer32). This last one requires a "base counter" which provides the rate to calculate the average.

When you create a performance counter category, you need to specify whether it is single-instance or multi-instance. MSDN helpfully explains that you should choose single-instance if you want a single instance of this category, and multi-instance if you want multiple instances. 🙂 However, I found a blog entry from the WMI team that actually explains the difference – it is whether there is a single-instance of the counter on the machine (for Azure, virtual machine) or multiple instances; for example, anything per-process or per-thread is multi-instance since there is more than one on a single machine. As you can see, since I expect multiple worker thread role instances, I made my counters multi-instance.

After creating the counters, you need to access them in your code. I created private members of my worker thread role instance:

Note that I use "." for machine name, which just means current machine, and I use the CurrentRoleInstance.Id from the RoleEnvironment to distinguish the instance of each counter. If instead you wanted to aggregate these across the role rather than per-role instance, you could just use RoleEnvironment.CurrentRoleInstance.Name.

Using the Custom Performance Counters

Once you’ve created the performance counters, you now want to access them in your code – for instance, in the part of your worker role where you are actually doing work to track the time it takes to process a unit of work. As you know, Azure calls the Run method of your role to do the work, and this method should not return – it instead runs an infinite loop waiting for work to do and then doing it. Let’s look at some simple code to use the performance counters defined above:

_TotalOperations

# worker thread operations executed

_OperationsPerSecond

# worker thread operations per sec

_AverageDuration

average time per worker thread operation

_AverageDurationBase

average time per worker thread operation base

Here’s the code for Run that uses these; note that it uses the System.Diagnostics Stopwatch class which provides access to a higher-accuracy timer than simply calling DateTime.Ticks(see this blog post for more information)

public override void Run()
{
while (true)
{
Stopwatch watch = new Stopwatch();
// Test querying the perf counter. Take a sample of the number of worker thread operations at this point.
CounterSample operSample = _TotalOperations.NextSample();
// Increment worker thread operations and operations / second.
_TotalOperations.Increment();
_OperationsPerSecond.Increment();
// Start a stop watch on the worker thread work.
watch.Start();
// Do the work for this worker
DoWork();
// Capture the stop point.
watch.Stop();
// Figure out average duration based on the stop watch, then increment the base counter.
_AverageDuration.IncrementBy(watch.ElapsedTicks);
_AverageDurationBase.Increment();
// Here's how you calculate the difference between a sample taken at the beginning and a sample at this point.
// Note that since this is # of operations which is monotonically increasing, this is kind of a boring use of
// a performance counter sample.
Trace.WriteLine("Worker Thread operations performance counter: " +
CounterSample.Calculate(operSample, _TotalOperations.NextSample()).ToString());
}
}

This code also shows how to take two samples of a counter and calculate the difference – the counter I’m showing is rather uninteresting, but it illustrates the approach.

Monitoring the Performance Counters

OK, so you’ve added updating the performance counters in your code, but now you want to be able to see the values while your Azure process is running. The Azure Diagnostics Monitor collects this information in a local cache but by default does not persist it anywhere unless you ask it to. You can do this by adjusting the DiagnosticMonitorConfiguration with code like this (note this is from my role OnStart method):

Once you’ve transferred the performance counter data, they go into the Azure table storage for the storage account you’ve passed (i.e. the account specified by DiagnosticsConnectionString in ServiceConfiguration.cscfg). You can then either use a table storage browser to look at WADPerformanceCountersTable or you can use the very helpful Windows Azure Diagnostics Manager from Cerebrata Software (there is a free 30-day trial). This tool reads the raw data from the Azure table storage and allows you to download it, graph it, etc.

References

Thanks to the following which provided invaluable information along the way to figuring this out: