AppInsights Analytics in the Real World

Ever since Application Insights (AppInsights) was released, I’ve loved it. Getting tons of analytics about site usage, performance and diagnostics – pretty much for free – makes adding Application Performance Monitoring (APM) to you application a no-brainer. If you aren’t using AppInsights, then you really should be.

APM is the black sheep of DevOps – most teams are concentrating on getting continuous integration and deployment and release management, which are critical pillars of DevOps. But few teams are taking DevOps beyond deployment into APM, which is also fundamental to successful DevOps. AppInsights is arguably the easiest, least-friction method of quickly and easily getting real APM into your applications. However, getting insights from your AppInsights data has not been all that easy up until now.

Application Insights Analytics

A few days ago Brian Harry wrote a blog post called Introducing Application Insights Analytics. Internally, MS was using a tool called Kusto to do log analytics for many systems – including Visual Studio Team Services (VSTS) itself. (Perhaps Kusto is a reference perhaps to the naval explorer Jacques Cousteau – as in, Kusto lets you explore the oceans of data?) MS then productized their WPF Kusto app into web-based Application Insights Analytics. App Insights Analytics adds phenomenal querying and visualizations onto AppInsights telemetry, allowing you to really dig into the data AppInsights logs. Later on I’ll show you some really simple queries that we use to analyze our usage data.

Brian goes into detail about how fast the Application Insights Analytics engine is – and he should know since they process terrabytes worth of telemetry. Our telemetry is nowhere near that large, so performance of the query language isn’t that big a deal for us. What is a big deal is the analytics and visualizations that the engine makes possible.

In this post I want to show you how to get AppInsights into a real world application. Northwest Cadence has a Knowledge Library application and in order to generate tracing diagnostics and usage telemetry, we added AppInsights. We learned some lessons about AppInsights on the way, and here are some of our lessons-learned.

Configuring AppInsights

We have 4 sites that we deploy the same code to – there are 2 production sites, Azure Library and Knowledge Library, and each has a dev environment too. By default the AppInsights key is configured in ApplicationInsights.config. We wanted to have a separate AppInsights instance for each site, so we created 4 in Azure. Now we had the problem of where to set the key so that each site logs to the correct AppInsights instance.

Server-side telemetry is easy to configure. Add an app setting called “AIKey” in the web.config. In a startup method somewhere, you make a call to the Active TelemetryConfig:

You can see how we’re using Razor syntax to get the AIKey static property value for the instrumentationKey value.

The next thing we wanted was to set the application version (assembly version) and site type (either KL for “Knowledge Library” or Azure for “Azure Library”). Perhaps this is a bit overkill since we have 4 separate AppInsights instances anyway, but if we decide to consolidate at some stage we can do so and preserve partitioning in the data.

Setting telemetry properties for every log entry is a little harder – there used to be an IConfigurationInitializer interface, but it seems it was deprecated. So we implemented an ITelmetryInitializer instance:

Now the version and siteType properties are added to every server-side log. Of course we could add additional “global” properties using the same code if we needed more.

Tracing

Last week we had an issue with our site – there’s a signup process in which we generate an access code and customers then enter the access code and enable integration with their Azure Active Directory so that their users can authenticate against their AAD when logging into our site. Customers started reporting that the access code “wasn’t found”. The bug turned out to be the fact that a static variable on a base class is shared across all child instances too – so our Azure Table data access classes were pointing to the incorrect tables (We fixed the issue using a curiously recurring generic base class – a study for another day) but the issue had us stumped for a while.

Initially I thought, “I can debug this issue quickly – I have AppInsights on the site so I can see what’s going on.” Turns out that there wasn’t any exception for the issue – the data access searched for an entity and couldn’t find it, so it reported the “access code not found” error that our customers were seeing. I didn’t have AppInsights tracing enabled – so I immediately set about adding it.

First, you install the Microsoft.ApplicationInsights.TraceListener package from NuGet. Then you can pepper your code with trace calls to System.Diagnostics.Trace – each one is sent to AppInsights by the TraceListener.

We decided to create an ILogger interface and a base class that just did a call to System.Diagnostics.Trace. Here’s a snippet:

The TraceInformation and TraceError methods are pretty straightforward – the TraceCustomEvent was necessary to enable custom telemetry. Using the logger to add tracing and exception logging is easy. We inject an instance of our AppInsightsLogger (more on this later) and then we can use it to log. Here’s an example of our GET videos method (we use NancyFx which is why this is an indexer method):

Custom Telemetry

Out of the box you get a ton of great logging in AppInsights – page views (including browser type, region, language and performance) and server side requests, exceptions and performance. However, we wanted to start doing some custom analytics on usage. Our application is multi-tenant, so we wanted to track the tenantId as well as the user. We want to track each time a user views a video so we can see which users (across which tenants) are accessing which videos. Here’s the call we make to log that a user has accessed a video:

Analytics Queries

Now that we’re getting some telemetry, including requests and custom events, we can start to query. Logging on to the Azure Portal I navigate to the AppInsights instance and click on the Analytics button in the toolbar:

That will open the AppInsights Analytics page. Here I can start querying my telemetry. There are several “tables” that you can query – requests, traces, exceptions and so on. If I want to see the performance percentiles of my requests in 1 hour bins for the last 7 days, I can use this query which calculates the percentiles and then renders to a time chart:

I love the way that the customDimensions (which is just a json snippet) is directly addressable. Here’s what the json looks like for our custom events:

You can see how the “siteType” property is there because of our ITelemetryInitializer.

Visualizations

After writing a couple queries, we can then add a visualization by adding a render clause. You’ve already seen the “render timechart“ above – but there’s also piechart, barchart and table. Here’s a query that renders a stacked bar chart showing user views (per tenant) in hourly bins:

This is just scratching the surface, but I hope you get a feel for what this tool can bring out of your telemetry.

Exporting Data to PowerBI

The next step is to make a dashboard out of the queries that we’ve created. You can export to Excel, but for a more dynamic experience, you can also export to PowerBI. I was a little surprised that when I clicked “Export to PowerBI” I got a text file. Here’s the same bar chart query exported to PowerBI:

Ah, so I’ll need PowerBI desktop. No problem. Download it, open it and follow the helpful instructions in the comments at the top of the file:

Now I can create visualizations, add custom columns – do whatever I would normally do in PowerBI.

One thing I did want to do was fix up the nasty “tenantId”. This is a guid which is the Partition Key for an Azure Table that we use to store our tenants. So I just added a new Query to the report to fetch the tenant data from the table. Then I was able to create a relationship (i.e. foreign key) that let me use the tenant name rather than the nasty guid in my reports:

Here’s what the relationship looks like for the “Users Per Tenant Per Hour Query”:

Once I had the tables in, I could create reports. Here’s a performance report:

One tip – when you add the “timestamp” property, PowerBI defaults to a date hierarchy (Year, Quarter, Month, Day). To use the timestamp itself, you can just click on the field in the axis box and select “timestamp” from the values:

Here’s one of our usage reports:

And of course, once I’ve written the report, I can just upload it to PowerBI to share with the team:

Look ma – it’s the same report!

Conclusion

If you’re not doing APM, then you need to get into AppInsights. If you’re already using AppInsigths, then it’s time to move beyond logging telemetry to actually analyzing telemetry and gaining insights from your applications using AppInights Analytics.

Happy analyzing!

7 Comments

Noam Ben Zeev

April 1, 2016 11:06

Hi Colin,Great coverage of the end to end scenario!As for your comment re the order by, it is indeed a bug in the Web UI grid and a fix should be available during the upcoming week.Thank you for your post,Noam

Michel van den Berg

May 11, 2016 10:11

How do you guys handle analytics over multiple insights instances? When do you decide that you need multiple instances and when not?

For example, you wrote that you are using 4 insights instances. Do you all hook them up to the same Power BI instance? Does that work? From what I can see the analytics explorer is per instance, not over multiple instances. Do you combine the analytics in Power BI? Or am not doing the correct analytics if I want to combine?

Another example, would be a load balanced app. Would definitely want that combined in one graph. Perhaps one insights instance per load balanced service would suffice?

A third example would be a 'processing' service, where the service contains of a web app and an independent (running in its own web app) webjob (to make it more scalable). Would you also log it to two instances?

There are several ways to handle multiple instances in AppInsights. Distinct applications should of course be in separate AppInsight instances. However, if you want to track a single application that is load balanced, then all 4 should be using the same AppInsights instance. You can add a ITelemetryInitializer (as we did above) to add the local machine name into all the telemetry sent from each web app. Then you can segregate all the telemetry by that dimension.