Monitoring Azure Media Services Application

Monitoring is a critical component of a production online application. A properly implemented monitoring system would enable you to receive alerts for key events and react appropriately. In this post, I will go over some sample code that I wrote to monitor a Media Services account. You can use the sample code as a starting point to start planning your own custom monitoring system for your application built on top of Media Services.

Sample Code

At a high level, the sample code provided works on the principle of executing some code against the service at regular intervals. The results of the operations are then logged and another piece of code checks the logs for failures at a regular frequency. If the programmed failure condition is triggered, an alert is fired. Given this, the code is divided in to two Visual Studio projects. The first project calls Media Services asset create and delete APIs at a regular interval and logs the results in an Azure table. The second project checks the top 5 entries in the Azure table at a regular interval to check for failures and fires an alert if all 5 entries were logged as failures. The App.config for the first project is as follows

Replace the <MediaAccountName> and <MediaAccountKey> with the name and key of the Media Services account you want to monitor. Likewise, replace the <StorageAccountName> and <StorageAccountKey> with the name and key of the storage account where you want the monitoring data to be logged The code that performs monitoring is as follows. As mentioned earlier, the code runs in an infinite loop and every minute it calls the Media Services APIs to create an asset and then delete it. If the API calls fail it logs a failure entry in an Azure table else it logs a success entry. Description of the functions used are below the code itself.

AssetLogEntity – This class defines the table entities. The code above logs a Status of 0 for success and a Status of 1 for failure. The FailureData string is used to write out a failure messages when Status = 1.

main – The main function creates the Media Services account context. It then creates the MonitoringData Azure table if it doesn’t already exist. Lastly it calls the Monitor function.

Monitor – This is the main monitoring code. The function runs in an infinite while loop. Every minute it uses the Media Services account context to create an asset and then delete it. If the APIs succeed it logs a success else it logs a failure by calling the LogMonitoringData function.

LogMonitoringData – This function performs the actual task of writing the data to the MonitoringData table. In order to make sure that the latest log items are at the top, it uses a rowkey that is calculated by subtracting the max DateTime value from the current DataTime. This is because Azure table enumerates items in ascending order.

The config file is similar to the first project except that it doesn’t have the Media Services account credentials (as this project focuses only on the entries in the MonitoringData table). At a high level, this code runs in an infinite loop and every minute it checks to see if there were 5 consecutive failure entries logged. If yes, then it raises an alert. The code is below and it is followed by a brief description of the functions in the code.

main – The main function simply creates a reference to the MonitoringData table and calls the CheckforFailures function.

CheckforFailures – This function loops through an infinite loop. Every minute it gets the top 5 entries in the MonitoringData table and checks to see if all the 5 entries were failures. If yes, then it raises an alert by printing out a message on the screen.

In order to simulate a failure, I regenerated the keys of the Storage account associated with the Media Services account by going to the Azure Management Portal. This action resulted in the Media Services account being unable to access the Storage account for creating the containers for an asset, thus resulting in failure of the CreateAsset call in the first project. After waiting for more than 5 minutes, I went back to the Azure Management Portal and synchronized the storage keys by clicking on “synchronize primary key” button (see screenshot below). This action resulted in the Media Services account being functional again after a few minutes. Below is a screenshot of the MonitoringData table entries. I have highlighted the entries corresponding to the failures. Below are screenshots of the console windows associated with the two sample code projects above. Both screenshots have the failure duration outlined.

Considerations

A few closing remarks on this topic

I did not use the storage account associated with the Media Services account to create the MonitoringData table. In fact, I used a storage account in a different datacenter. The sample above simulated one failure situation that was related to the Media Services account itself but if you want to catch failures related to networking issues or storage services issues in the datacenter where your media application is hosted, then its best that you have your monitoring data being written in another location. The same applies to where you run your monitoring and alerting code. I ran the two sample codes above from outside the datacenter where the Media Services account was located.