Microsoft Azure Stack is an extension of Azure – bringing the agility and innovation of cloud computing to your on-premises environment and enabling the only hybrid cloud that allows you to build and deploy hybrid apps anywhere. We bring together the best of the edge and cloud to deliver Azure services anywhere in your environment.

Auditing Media Assets Lifecycle – Part 1

Media applications dealing with high value content are typically required to abide by MPAA, CDSA or equivalent compliance requirements. As part of the audit process associated with them, you will likely be asked to produce an audit report that demonstrates the lifecycle of Media Assets as it propagates through your applications and services. In this blog (which is divided in to multiple parts), I will cover how you can generate an audit report for your Media Assets as they flow through Media Services. Part 1 of this blog will focus on enabling you to create an asset audit report that shows when Media Assets were created and deleted.

Media Assets in Media Services

When you create a Media Asset, Media Services generates a GUID and uses that GUID to create a Media Asset Id. The Media Asset Id is prefixed by “nb:cid:UUID:” followed by the GUID. These ID’s are in URN format where the “nb” actually stands for Media Services codename internally (Nimbus), and the cid stands for Content ID. In other words, the Media Asset Id takes the form of “nb:cid:UUID:<GUID>”. Media Services then creates a record for the asset and stores it internally. Media Services also creates a container named “asset-<GUID>” in the specified Storage Account. Once the Asset is created, you can upload media files in the storage container. When you delete a Media Asset, Media Services deletes the asset record from its internal database and also deletes the storage container. Given this you can use the Media Services APIs to determine the create time of an Asset as long as it has not been deleted, but there is no way to determine the delete time of an Asset unless you kept track of it in your media application.

Tracking creation and deletion of Media Assets via Storage logs

Since a media asset is represented as a container in Storage, you can use the Storage logs to determine creation and deletion time of Media Assets. In order to do that it is necessary that you have Storage logging enabled in your Storage account. See How to: Configure logging to learn about this. Note that the retention policy you choose will dictate your ability for how far back you will be able to go in your audit report. If you choose zero then your logs will not be deleted and you will have the ability to go as far back as the date you enabled logging. Azure Storage saves the logs in the storage account in a container called $logs. More details about how logs are stored and log naming convention can be found in the Storage Analytics Logging MSDN page.

Sample Code

The sample code provided below uses both the Media Services Assets Collection and the Storage logs to generate an Azure Storage Table called AssetAudit. This table can be used for generating an asset audit report showing the creation and deletion time of assets. At a high level, the logic is as follows.

The code enumerates through all the assets using the Media Services API.

For each enumerated asset, it uses the Asset.Created property to create a table entry in the AssetAudit Table.

In the above App.Config, replace <MediaAccountName> and <MediaAccountKey> with your Media Services Account Name and Key. Also replace <StorageAccountName> and <StorageAccountKey> with the name and key of the storage account associated with your Media Services account.
The code is as follows

A brief description of the functions in the code above is as follows
ProcessAssetData
This function loops through all the assets in the provided Media Services Account. By default, Media Services returns 1000 assets in Assets collection. The function makes use of Skip and Take to make sure that all assets are enumerated (in case you have more than 1000 assets in your account).
ParseStorageLogs
This function enumerates all the blobs under $logs/blob and saves the processed blob as the last log file processed so that they are not re-processed if the code is run repeatedly.
SaveLastLogFileInConfig
This function saves the last processed log file name in App.Config so that it can be retrieved if the program is rerun.
GetBlobData
This function downloads the blob from Storage and reads the content in to a string.
ParseDelimitedString
This function parses a string based on provided delimiter. The parsed data is returned as a string collection.
ParseLogLine
This function parses each log line to extract out the CreateContainer and DeleteContainer operations for containers that start with “asset-“.
InsertAssetData
This function adds an entry to the AssetAudit table.

Asset Audit Data

Once you run the code above, the AssetAudit table will be created. Below is a screenshot of the contents of this table against a test account that I used. I have highlighted a matching pair of Create and Delete for an Asset. These entries could only be captured from the Storage logs when the code above was run as Media Services had no entry for it anymore
You can also use Excel Power Query to load the above table data in to Excel. With Excel you can do additional filtering or load the above in to a Pivot table for additional analysis. If you have never used Excel Power Query, you can download it from “Download Microsoft Power Query for Excel” web page. Once installed you can start Excel and you will see a tab called “POWER QUERY”. Click on that tab and then click on “From Other Sources” button and you will see a menu item called “From Windows Azure Table Storage” as shown in the screenshot below
To import the data from AssetAudit table, select the menu item above and follow the instructions. Once the tables are loaded in the “Navigator” pane on the right hand side, you can double click on the AssetAudit table and a new window will open up. A screenshot of that window is as follows.
Click on the button next to the column labeled “Content” and then click OK. After that you can click on “Apply & Close” button on the top. This will close the current window and load the Table data in Excel. Now you can use Excel to analyze the data in the way you see fit

Considerations

Finally, please note the following as you consider using this sample code for your application

The sample code provided in this blog is designed to work with a Media Services account that has all assets in a single storage account but it can be easily adapted to work with multiple storage accounts.

The audit is limited to the retention policy associated with the Storage logs.

When you run the sample in the debugger, the App.Config file will not be updated with last log blob that was processed. You will see that happen only when you run the sample outside the debugger.

The exceptions are only printed to the Console. You can write those out to an Azure Table or a local file to see the errors (if any).