Microsoft development, Azure and more fun stuff!

Recent Updates

I recently had the opportunity to assist on a project where a partner was using N-Series Azure VMs. My part of this effort was developing a script to automate the setup of the VMs. To perform the VM setup and configuration, an ARM template was used. The ARM template approach was used because doing so provided consistency with several other ARM templates being used for other parts of the project.

Setting up Azure VMs using ARM templates is common. There are many articles, blog posts, and sample templates available to help get started. That isn’t itself especially interesting. The interesting part, at least for me, was the N-Series aspect. N-Series VMs require a separate step to install the NVIDIA driver to take advantage of the GPU capabilities of the VM. There are instructions on how to install the driver, but those instructions assume you like to remote into the VM each time you create a VM, and then run an installation program. That’s tolerable if doing it only a few times. Any more than that, and it’s time for automation.

The v370.12 driver (which is the current version linked via the Azure documentation page) uses a self-extracting file to first extract the setup components to a directory, and then executes the setup program. By scouring a few other blogs on performing a silent install of NVIDIA drivers, I could piece together the necessary switches to provide to the installation program to perform a silent install.

This tells the installation program to install silently, to not perform a reboot after the installation is complete, and to perform a clean install (restores all NVIDIA settings to the default values).

Now I need to work that it into a PowerShell script to execute via a custom script extension. By doing so, I can let ARM do its thing by provisioning the VM and related resources (NIC, Virtual Network, IP address, etc.), and then invoke a PowerShell script to install the NVIDIA driver.

The custom script extension will execute a few different steps:

Download the NVIDIA driver setup file from Azure Blob storage. I put the setup file in blob storage to make sure that this specific one is the one to be used.

Download a PowerShell script which will execute the NVIDIA driver setup program with parameters to do so silently.

Wait for the installation program to finish

Force a reboot of the VM

It should be noted that the driver installation and GPU detection can take a couple of minutes.

As you can see in the following snippets, the custom script extension and related PowerShell script are fairly trivial.

In this scenario, I also need to get the assets used by the custom script extension – the NVIDIA driver setup file and PowerShell script (which will execute the NVIDIA driver setup file) – uploaded to Azure Blob storage. That can easily be accomplished with the same PowerShell script used to deploy the ARM template. That script will perform the following tasks:

Create a new resource group

Create a new storage account and container

Upload the NVIDIA driver setup file and related PowerShell script to the newly created storage account

An earlier version of the NVIDIA driver, v369.95, was not digitally signed. It was also provided as a ZIP file instead of an EXE (like v370.12). To use this version of the NVIDIA driver, a few changes to the setup script are necessary. First, the file contents need to be extracted/unzipped. That’s doable via some PowerShell in the script executed via the custom script extension. Getting around the lack of a digitally signed driver is a bit more . . . interesting. If you were to install the driver manually, you would receive a prompt from Windows asking you to confirm that installing the driver is REALLY what is desired.

Completing the manual installation will result in a certificate installed to the VM’s Trusted Publisher certificate store. The certificate can then be exported and saved to Azure Blob storage.

I can use that certificate as part of the automated install process. By using the certutil.exe program it is possible to install the certificate into the Trusted Publisher store on a new VM. This step can be included in the PowerShell script executed via the custom script extension.

An alternative approach is to create a custom VM image with the necessary NVIDIA driver already installed. The advantage with this approach is you don’t have to go through the custom script step. However, any new VM deployed from such an image will still need to go through a reboot after GPU detection following the first startup. You can also add additional software or configuration as needed. The disadvantage is you’re then accepting responsibility for keeping the VM patched on a regular basis. If you use an image provided by Microsoft, those images are patched on a regular (often at least once per month) basis.

Resources

Here are some resources which helped me in coming up with the solution presented above.

Share this:

Like this:

Introduction

Azure Managed Disks were made generally available (GA) in February 2017. Managed Disks greatly simplify working with Azure Virtual Machines (VM) and Virtual Machine Scale Sets (VMSS). They effectively eliminate the need for you to have to worry about Azure Storage accounts and related VHD constraints/limits. When using managed disks for VMs or VMSS, you select the type of disk storage (SSD or HDD) and the size of disk needed. The Azure platform takes care of the rest. Besides the simplified management aspect, managed disks bring several additional benefits, but I’ll not reiterate those here, as there is a lot of good info already available (here, here and here).

While managed disks simplify management of Azure VMs, they also simplify working with VM images. Prior to managed disks, an image would need to be copied to the Storage account where the derived VM would be created. Doable, but not exactly convenient. With the introduction of managed disks, since the concept of using Storage accounts for disks and images has gone away, there is no need to copy the image. You can now create managed images as the ARM resources. You can easily create a VM by referencing the managed image, so long as the VM and image are in the same region and the same Azure subscription. You can consult the following two articles for detailed documentation on this topic:

However, what if you need to use the managed image in another Azure subscription (to which you have access)? Or, what if you need to use the managed image in another region? These capabilities are not yet available as part of the platform. However, there are workarounds you can use, with the currently available capability, to facilitate these needs.

In this post, we’ll explore the following two common scenarios:

Copy a managed image to another Azure subscription

Copy a managed image to another region

High Level Steps

To get a managed image in one Azure subscription to be available for use in another Azure subscription, there are a series of steps that currently need to be followed. In the near future, I expect this process to greatly be simplified by enhancements to the Azure platform’s managed image functionality. Until then, the high-level steps are as follows:

In the target subscription, create an managed image from the copied snapshot

Optional: from the new managed image in the target subscription, create a new temporary VM

Delete the snapshot in both the source and target Azure subscription

Delete the temporary VM created in step #8

Getting Started

For the purposes of this post, I’m going to assume you have already created a VM using managed disks, configured it to your liking (e.g. installing some software, making configuration changes, etc.), and generalized the VM.

Create an Image

Assuming you have a generalized (deallocated) VM, the next step is to create a managed image. It is worth pointing out that, at this time, creating the image is largely irrelevant when trying to copy the image to another region and/or subscription. As you’ll soon see, the artifact that is copied is the snapshot of disk(s) of the source (generalized) VM. The ability copy the image is not yet supported . . . hence this blog post to describe a workaround.

If you already have the Image, you can obviously skip this step. The steps are as follows:

Create a snapshot

Now that you have an image, the next step is to create a snapshot of the OS disk of the source VM. If your image needs data disks, you’ll want to create a snapshot of the data disks as well (not shown below).

Copy the snapshot

The next step is to copy the snapshot to the target Azure subscription. In the following example, the first thing to do is grab the snapshot’s Resource ID. That ID is used to specific the source snapshot when creating the new snapshot.

Alternative: Copy the snapshot to a different region for the same subscription

The previous examples showed how to copy the snapshot to a different subscription, with the restriction being the region for the source and target must be the same. There may be times when you need to get the snapshot to another region. The follow example shows how to copy the snapshot to another region, yet under the context of the same Azure subscription. The big difference here is the need to get at the blob which is the basis for the snapshot. That can be accomplished by getting a Shared Access Signature (SAS) for the snapshot.

Alternative: Copy the snapshot to a different region for a different subscription

The previous example showed how to copy the snapshot to a different region, yet associated with the same subscription. In the following example, we’ll tweak the example script a bit to show copying the snapshot to a different region and a different subscription.

These three examples should cover the scenarios needed to get the snapshot wherever it needs to be. From there, the steps to create the image should be the same, since they all start with the snapshot.

Create an Image (in target subscription)

Once the snapshot has been copied to the target Azure subscription, the next step is to use the snapshot as a basis for creating a new managed image. Be sure to proceed to the next step (Create a temporary VM from the Image) don’t stop here!

Earlier, when you created the snapshot and copied it to the target Azure subscription, you may have noticed the process went relatively quick. One reason for this is how Azure copies the data – it uses a copy-on-read process. Meaning, the full dataset isn’t copied until it is needed. To trigger the data to be fully copied, a VM can be created. The VM created in this step is used to trigger the data transfer, and then we can safely delete the VM and snapshots. This step can be considered optional – as the first time a VM is created from the snapshot, doing so will ensure the data is fully copied.

In the example below, I’m using an ARM template to provision the new (temporary) VM. This template is very similar to this one, expect that I’ve modified the one I’m using to allow for the use of a managed image.

Delete the snapshots

Since the new image and temporary VM have been created in the target subscription, there is no longer a need for the snapshot. You would want to delete the snapshot in both the source and target Azure subscription.

Delete the temporary VM

Earlier there was a step to create a temporary VM. The VM was created to trigger the data copy process. It serves no other purpose at this point, thus it is safe to delete. If you followed the earlier steps to create the temporary VM, it was created in its own resource group. Thus, simply delete the resource group.PowerShell

Remove-AzureRmResourceGroup -Name $rgNameTemp -Force

Azure CLI 2.0

az group delete -n $resourceGroupName

Summary

As you can see, there are several steps necessary to copy a Managed Disk Image from one Azure subscription to another. The steps aren’t difficult, just a bit unintuitive at this point. This post should help make the process a bit easier to understand. When the ability to copy images across subscriptions and regions is available as a first-class feature in Azure (hopefully later this year), this post will be effectively obsolete. I’m OK with that. 🙂

Share this:

Like this:

Recently I had the opportunity to do a little (and I mean “little” in the most literal sense) Office add-in development. If you follow me on this blog or Twitter, you know that I work primarily in the Azure space. Building an Office add-in is certainly new to me, but I figured it would be kind of fun to learn about something new. This journey started as a result of a partner that I worked with who was having problems getting their add-in to load correctly in both Word Online and Word (on their desktop). Odd . . . right? It should work the same when hosted in both Office Online apps and Office.

I knew from prior reading, that the “new” Office add-in model was (to likely over simplify) essentially creating a web application that would be hosted within an Office desktop (e.g. Word 2016) or Office online (e.g. Word online) application. If the add-in needed to interact with the Word, Excel, Outlook environment in any way, there is a JavaScript API to do so. Seems easy enough.

To get started, I did a little Binging (just sounds odd . . . but whatever) for Office add-in development. I wanted to start with “official” Microsoft documentation, so I ignored results from StackOverflow.com and blogs (wait . . what if it is my blog . . .this blog . . .) One of the top results was the “Office Add-ins platform overview” page at https://dev.office.com/docs/add-ins/overview/office-add-ins. Sweeta!

Reading . . .reading . . . reading

Finally, I get to something that looks like code . . .well at least a picture of code, at https://dev.office.com/docs/add-ins/overview/office-add-ins#anatomy-of-an-office-add-in. This section states that the minimal version of an add-in is a “static HTML page that is displayed inside an Office application, but doesn’t interact with either the Office document or any other Internet resource.” I can do that! I’ll simply write a “Hello World” app to just see it work, and then go from there. The page includes a picture of two components needed for a “Hello Word” Office add-in, the manifest file and the web page. Bam!

I created the most basic HTML page . . . which just happened to look exactly like the one in the picture. I then proceeded to read up on how to debug the add-in via sideloading the add-in via Office and Office Online. This is easy!

I started by sideloading the add-in via Word (desktop). Hot damn . . . it worked!!

This is where, as a developer, I start to question my life choices. I also realize there must be something else going on causing this funkiness. I tried the normal browser rotation game – Chrome, Edge, IE – all produced the same results. Hmmmm.

I decided to do some more reading. I found this page, https://dev.office.com/docs/add-ins/get-started/create-an-office-add-in-using-any-editor, which promised to give me a working sample. I followed it, and it worked. So, what was different (besides being a whole lot more code)? As I dug around the generated code, I noticed the home.html file contained two JavaScript references, one for jQuery and one for office.js. I could understand the jQuery one, but what’s this office.js thing? It must be the API used to interact with Office apps. Ok . . . but my “Hello World” app isn’t doing anything with Office apps. What gives?

“All pages within an Office Add-ins are required to assign an event handler to the initialize event, Office.initialize. If you fail to assign an event handler, your add-in may raise an error when it starts. Also, if a user attempts to use your add-in with an Office Online web client, such as Excel Online, PowerPoint Online, or Outlook Web App, it will fail to run. If you don’t need any initialization code, then the body of the function you assign to Office.initialize can be empty, as it is in the first example above.”

YES! YES!! That would have been good information a few hours ago . . . . as on that “Hello World” page with the simple HTML example code. Put some big, red blinky arrows around this, for Pete’s sake!

I proceeded to add the most basic office.js initialization code to my sample “Hello World” page (which just happens to be hosted on Azure Web Apps, with a local Git continuous deployment option . . . see, I had to get Azure in here somewhere).

I proceeded to reload my add-in in Word online. It worked! The simple console.log() statements showed up in the JavaScript console too. Yippie!!

That’s a lovely story . . .what’s the point?

If the documentation example seems too simple to be true, it probably is? Eh . . . sometimes. The point here is that for Office add-ins, you are absolutely required to reference office.js and at least set up an empty initialization block. Doing so helps Office in setting up some of the logical infrastructure needed for the add-in environment. It’s like magic. Only magic isn’t real. Mostly.

Like this:

One of the great features recently added to Azure SQL Database is the ability to authenticate to Azure SQL Database using Azure Active Directory. This provides an alternative to exclusively using SQL credentials. By leveraging Azure AD authentication, you can greatly simplify management of database permissions by continuing to use existing identities, as well as leveraging AD groups.

The article here does a decent job of explaining the basics of how Azure AD authentication with SQL Database works, and the steps needed to do so. One area that it doesn’t yet cover is obtaining an Azure AD authentication token and using that token to authenticate with SQL Database. Actually . . . it does, sort of. The article assumes a certificate will be used. That isn’t always the case. In the following sections, I will show you how to obtain an Azure AD authentication token for a user (in Azure AD directory), and use that token for authentication with SQL Database.

On the resulting Create blade, provide a friendly name, select an application type of Native, and provide a redirect URL (which is largely irrelevant in this scenario for a native console application)

Add Azure SQL Database to the list of APIs which will require permission from your application

This step is very important. You will need to add Azure SQL Database to the list of APIs / applications which will be granted delegated permission via your application. Failure to do this will result in an error message similar to the following:

“AADSTS65001: The user or administrator has not consented to use the application with ID ‘{your-application-id-here}’. Send an interactive authorization request for this user and resource.”

Before you can continue, you need to have followed the prerequisites steps stated at the top of this post. You especially need to be sure you have created an Azure AD contained database user. If you fail to do that, you will not see “Azure SQL Database” in the list (as specified below).

Setting the permission is fairly easy via the Azure portal.

Select the newly created application (in this case, it was ContosoConsole6)

On the Settings blade, select Required permissions.

Add a new required permission and select Azure SQL Database as the API. You’ll want to search for “azure” to get “Azure SQL Database” to appear in the list.

Be sure to select the checkbox for “Access Azure SQL DB and Data Warehouse”.

I should point out that even after adding the delegated permission as shown above, you will still get the previously mentioned error. We’ll solve that next.

Consent to allow your application to access Azure SQL Database

That error you received before about the administrator having not consented to use application is something we need to get past. To do so, you can force a one-time consent dialog so you can consent to your application delegating access to Azure SQL Database. You’ll need to craft a URL in the following format (wrapped for readability):

Since you’ll be working with Azure AD, you’ll want to use ADAL to make getting the Azure AD authentication token easy.

Add code to obtain an Azure AD authentication token

Finally! Some code! First, in my example, I set up a few constants which represent information about Azure AD and the resource for which I want to obtain an authentication token. For SQL Database, the resource is https://database.windows.net/.

You’ll need a client ID as part of the call to AcquireTokenAsync(). The client ID is the Application ID for your app registered in Azure AD. The Azure AD tenant value is the friendly name for your Azure AD tenant (e.g. contoso.onmicrosoft.com). Thanks to using ADAL, the code to get the authentication token is very easy – just two lines of code:

You’ll notice the above code calls a method GetUserCredential() to obtain a UserCredential object. This represents the Azure AD user for which to obtain the authentication token. In this example, the user details are hard coded. Yes, this is bad. I’m showing it here strictly as an example. In future posts, I’ll show a few other ways to (better) handle the user details.

If all goes well, after executing the AcquireTokenAsync() method you should receive an Azure AD (JWT) authentication token as part of the resulting AuthenticationResult object. For fun, paste it in at https://jwt.io to decode it. You should get something similar to the screenshot below.

The following code to query the database is ultra-simple. I just want to make sure the connection is successful and I can execute a basic command. This command will return the name of the connected user . . . which should correspond to that of the specified Azure AD user.

As you can hopefully see, the steps to using an Azure AD authentication token with Azure SQL Database are not especially complicated. There are a few minor hoops to jump through, but nothing too serious.

It is worth pointing out that this solution does take a dependency on Azure AD. In the unlikely event that Azure AD is unavailable, then you may be unable to access your database resources using your Azure AD credential. I think it is a wise strategy to also keep a few SQL identities in place and have a process ready to use those – just in case. After all, designing robust applications for the cloud is about designing for failure. Always be prepared.

The first edition of the book was relatively popular, with over 200,000 copies either downloaded or distributed in print. Thank you for the amazing support! I’m very happy so many people found the book to be helpful.

When the Microsoft Press editor that worked on the first book initially approached me about working on an update, I was honestly a bit hesitant. Writing a technical book is a lot of work . . . especially a book on a platform such as Azure that seems to be constantly evolving. However, after many discussions with Robin, and my wife (getting spousal approval was very important as this effort would require many nights and weekends away from my family, which added a new baby to the party earlier this year), I decided to write the update. Thankfully Robin agreed to join in on the fun too. She’s a lot of fun to work with!

My thinking going into this process was there would be a few significant updates, but mostly it would be changing a few screenshots, updating a product name or two (i.e. Azure Web Sites became Azure Web Apps), and maybe including a bit of content around new services like Azure Service Fabric. It wouldn’t be that much work. Wrong!

Azure is a fast-moving platform. In the nearly 13 months since the first edition of the book was released to when I got started on the second edition, many things had changed. The second edition contains quite a few significant updates. The sections on Azure Web Apps, Azure Virtual Machines, Virtual Networks, Storage, and management tooling all received major edits. Key concepts such as Azure Resource Manager, use of the (new) Azure Portal, and the (almost entire) removal of Azure Cloud Services also received increased focused in the second edition. We also added a new chapter on Additional Azure Services which provides a brief introduction to several other Azure services Robin and I felt were important to understand. There are so many valuable services in the Azure platform, but we couldn’t discuss all of them in the book (or else we still might be working on the book, and it would be huge). These are services we personally felt were of significance for the broad majority of people. (Robin adds more info on the changes in her comment here.)

I would be remiss if I didn’t once again thank the Azure experts that volunteered their precious time to help review the book. Many of these people helped on the first edition too. Their expertise and honest feedback was critical in writing this book. Thank you!

Share this:

Like this:

Last month I had the privilege to speak at one of the many excellent technology conferences in the Midwest, StirTrek. It’s always fun to speak, and attend, StirTrek. Not only are the sessions great, but afterwards there is a showing of one of the summer’s biggest movies. This year the movie was Captain America: Civil War.

My session this year was “Work on Your ARM Strength”. The goal with the session was to provide some guidance on how to write Azure Resource Manager (ARM) templates. ARM templates are a key part of working with Azure, and it is important to understand how to write, debug, and deploy them.

StirTrek did an excellent job in recording this year’s sessions. You can view all the session on StirTrek’s YouTube channel. You can watch my “Work on Your ARM Strength” session here, or via the embedded video below. Enjoy!

There are many options for configuring monitoring and alerting for Azure resources. The Azure portal will show some default metrics. It is also possible to enable more advanced or custom diagnostic settings. The links below provide additional details on enabling diagnostics and monitoring for two popular Azure compute resources, Azure Web Apps and Azure Virtual Machines. Visual Studio Application Insights can also be leveraged to monitor the usage and performance of applications.

As previously mentioned, the Azure portal will show some default metrics for many Azure resources. For example, the screenshot below shows the monitoring tile for an Azure Web App, which has been configured to display three key values: Average Response Time, CPU Time, and Requests.

The metric values displayed on this tile are not retained indefinitely. For an Azure Web App, the retention policy is as follows:

Minute granularity – 24 hour retention

Hour granularity – 7 day retention

Daily granularity – 30 day retention

By using the Azure Insights API it is possible to programmatically retrieve the available default metric definitions (the type of metric such as CPU Time, Requests, etc.), granularity, and metric values. With the ability to programmatically retrieve the data, comes the ability to save the data in a data store of your choosing. For example, that data could be persisted to Azure SQL Database, DocumentDB, or Azure Data Lake. From there you could perform whatever additional analysis is desired.

It should be noted that the Azure Insights API is not the same as Application Insights.

Once the available metric definitions are known, it is easy to retrieve the required metric values. Use the metric’s name ‘value’ (not the ‘localizedValue’) for any filtering requests (e.g. retrieve the ‘CpuTime’ and ‘Requests’ metric data points). The request / response information for this API call do not appear as an available task at https://msdn.microsoft.com/en-us/library/azure/dn931930.aspx. However, it is possible to do so, and the request URI is very similar to that of listing the metric definitions.

For example, to retrieve just the Average Response Time and Requests metric data points for an Azure Web App for the 1 hour period from 2016-02-18 20:26:00 to 2016-02-18 21:26:00, with a granularity of 1 minute, the request URI would be as follows:

Using the REST API is very helpful in terms of understand the available metric definitions, granularity, and related values. This information can be very helpful when using the Azure Insights Management Library.

Retrieving Metrics via the Insights Management Library

Just like working with the REST API, there are three basic steps for working with the Insights Management Library:

Authenticate the Azure Insights request

Retrieve the available metric definitions

Retrieve the metric values

The first step is to authenticate by retrieving the JWT token from Azure AD. Assuming the Azure AD service principle is already configured, retrieving the token can be as simple as shown in the code sample below.

throw new InvalidOperationException(“Failed to obtain the JWT token”);

}

string token = result.AccessToken;

return token;

}

The primary class for working with the Insights API is the InsightsClient. This class exposes functionality to retrieve the available metric definitions and metric values, as seen in the sample code below:

To find the resource URI for a desired resource, one approach is to use the https://resources.azure.com tool. Simply browse to the desired resource and then look at the URI shown, as in the screenshot below.