Connect Azure Diagnostics to Event Hubs sink

By default, Azure Diagnostics always sends logs and metrics to an Azure Storage account. An application may also send data to Event Hubs by adding a new Sinks section under the PublicConfig / WadCfg element of the .wadcfgx file. In Visual Studio, the .wadcfgx file is stored in the following path: Cloud Service Project > Roles > (RoleName) > diagnostics.wadcfgx file.

In this example, the event hub URL is set to the fully qualified namespace of the event hub: Event Hubs namespace + "/" + event hub name.

The event hub URL is displayed in the Azure portal on the Event Hubs dashboard.

The Sink name can be set to any valid string as long as the same value is used consistently throughout the config file.

Note

There may be additional sinks, such as applicationInsights configured in this section. Azure Diagnostics allows one or more sinks to be defined if each sink is also declared in the PrivateConfig section.

The Event Hubs sink must also be declared and defined in the PrivateConfig section of the .wadcfgx config file.

The SharedAccessKeyName value must match a Shared Access Signature (SAS) key and policy that has been defined in the Event Hubs namespace. Browse to the Event Hubs dashboard in the Azure portal, click the Configure tab, and set up a named policy (for example, "SendRule") that has Send permissions. The StorageAccount is also declared in PrivateConfig. There is no need to change values here if they are working. In this example, we leave the values empty, which is a sign that a downstream asset will set the values. For example, the ServiceConfiguration.Cloud.cscfg environment configuration file sets the environment-appropriate names and keys.

Warning

The Event Hubs SAS key is stored in plain text in the .wadcfgx file. Often, this key is checked in to source code control or is available as an asset in your build server, so you should protect it as appropriate. We recommend that you use a SAS key here with Send only permissions so that a malicious user can write to the event hub, but not listen to it or manage it.

Configure Azure Diagnostics to send logs and metrics to Event Hubs

As discussed previously, all default and custom diagnostics data, that is, metrics and logs, is automatically sent to Azure Storage in the configured intervals. With Event Hubs and any additional sink, you can specify any root or leaf node in the hierarchy to be sent to the event hub. This includes ETW events, performance counters, Windows event logs, and application logs.

It is important to consider how many data points should actually be transferred to Event Hubs. Typically, developers transfer low-latency hot-path data that must be consumed and interpreted quickly. Systems that monitor alerts or autoscale rules are examples. A developer might also configure an alternate analysis store or search store -- for example, Azure Stream Analytics, Elasticsearch, a custom monitoring system, or a favorite monitoring system from others.

In this example, the sink is applied to logs and is filtered only to error level trace.

Deploy and update a Cloud Services application and diagnostics config

Visual Studio provides the easiest path to deploy the application and Event Hubs sink configuration. To view and edit the file, open the .wadcfgx file in Visual Studio, edit it, and save it. The path is Cloud Service Project > Roles > (RoleName) > diagnostics.wadcfgx.

At this point, all deployment and deployment update actions in Visual Studio, Visual Studio Team System, and all commands or scripts that are based on MSBuild and use the /t:publish target include the .wadcfgx in the packaging process. In addition, deployments and updates deploy the file to Azure by using the appropriate Azure Diagnostics agent extension on your VMs.

After you deploy the application and Azure Diagnostics configuration, you will immediately see activity in the dashboard of the event hub. This indicates that you're ready to move on to viewing the hot-path data in the listener client or analysis tool of your choice.

In the following figure, the Event Hubs dashboard shows healthy sending of diagnostics data to the event hub starting sometime after 11 PM. That's when the application was deployed with an updated .wadcfgx file, and the sink was configured properly.

Note

When you make updates to the Azure Diagnostics config file (.wadcfgx), it's recommended that you push the updates to the entire application as well as the configuration by using either Visual Studio publishing, or a Windows PowerShell script.

View hot-path data

As discussed previously, there are many use cases for listening to and processing Event Hubs data.

One simple approach is to create a small test console application to listen to the event hub and print the output stream. You can place the following code, which is explained in more detail
in Get started with Event Hubs), in a console application.

Troubleshoot Event Hubs sinks

The event hub does not show incoming or outgoing event activity as expected.

Check that your event hub is successfully provisioned. All connection info in the PrivateConfig section of .wadcfgx must match the values of your resource as seen in the portal. Make sure that you have a SAS policy defined ("SendRule" in the example) in the portal and that Send permission is granted.

After an update, the event hub no longer shows incoming or outgoing event activity.

First, make sure that the event hub and configuration info is correct as explained previously. Sometimes the PrivateConfig is reset in a deployment update. The recommended fix is to make all changes to .wadcfgx in the project and then push a complete application update. If this is not possible, make sure that the diagnostics update pushes a complete PrivateConfig that includes the SAS key.

I tried the suggestions, and the event hub is still not working.

Try looking in the Azure Storage table that contains logs and errors for Azure Diagnostics itself: WADDiagnosticInfrastructureLogsTable. One option is to use a tool such as Azure Storage Explorer to connect to this storage account, view this table, and add a query for TimeStamp in the last 24 hours. You can use the tool to export a .csv file and open it in an application such as Microsoft Excel. Excel makes it easy to search for calling-card strings, such as EventHubs, to see what error is reported.