thoughts on cloud native stuff

Deploy a hybrid Kubernetes Cluster to Azure Container Service

Lately, I have been working a lot with Kubernetes as one (of many) solutions to run Docker containers in the cloud. Microsoft therefore offers Azure Container Service (ACS), a service to create and (partly) manage a Kubernetes cluster on Azure.

Unfortunately, you cannot customize all the properties of the Kubernetes deployment with this approach, e.g. if you want to place the cluster in an existing Azure Virtual Network (VNET) or if you want to run multiple node types within the cluster to be able to run Linux and Windows based images/pods in parallel.

To achieve this, you must use the ACS engine which is a kind of “translator” between cluster configurations (which are provided in JSON format) and Azure Resource Manager templates.

ACS Engine

The ACS engine provides a convenient way to generate an ARM template that creates a Kubernetes cluster for you in Azure. The nice thing about it is, that you can influence a lot more properties of the cluster than you can do via the portal or CLI. But more on this later…

If you execute the ACS engine, the resulting template created consists of all the resources, you need to run a cluster in Azure, e.g.:

Availability Sets for Master and Agent nodes

VMs / VM extensions

NICs / VNET configurations

Load Balancer

etc.

You can deploy the ARM template as you would deploy any other template to Azure by running a Powershell or CLI command, even via the portal.

But let’s get to our sample, creating a hybrid Windows/Linux cluster…

Hybrid Cluster With ACS Engine

We will start by creating a cluster definition file…

Some details on that:

First of all, we create the cluster configuration, setting the Kubernetes version to “1.8” (starting line 4)

the profile of the master node is set, giving it a name and a VM type “Standard_D2_v2” (beginning line 8).

agent profiles are defined, setting up two profiles. One for Linux nodes, the other one for Windows nodes…also setting the VM size to “Standard_D2_v2” (lines 13 to 27)

next, the two profiles are configured, each with the corresponding access information (user name or ssh key), lines 28 to 41

the last step is to set the service principal (application id and password, lines 42 to 45), which is needed by Kubernetes to interact with Azure. E.g. when you define a service of type “LoadBalancer”, K8s reaches out to Azure to create an external IP at the Azure Load Balancer

the .env file has to be placed in the same folder as the JSON file

In case a SSH key has to be created, you can do this on Windows via PuttyGen or on Linux/Mac via ssh directly. The value of the public key must be specified in the definition file (keyData, line 37).

If no service principal already exists, you can create it using the following command:

The values from “appId” and “password” must be stored in the corresponding properties of the cluster configuration (lines 43 and 44).

Generate The ARM Template

In order to create the ARM template from the cluster configuration, we first need the ACS-engine binary. You can download it from here. Unzip the contents and place the destination folder into the PATH environment variable, to be able to run it from anywhere on your machine.

Next, run the ACS engine with your configuration…

acs-engine generate .\k8s-hybrid.json

The ACS engine takes the JSON file and generates an _output directory with all the necessary artifacts (among other things, a rather large ARM template + parameters-file) to deploy the K8s cluster to Azure. Here’s a visual representation of the ARM template:

ARMVIZ representation of the resulting template

Next, you simply deploy the template, as you would do it with other ARM templates. First, create a Azure Resource Group:

az group create -n k8scluster-rg -l westeurope

Afterwards, switch to the _output/[CLUSTERNAME] folder and deploy the template:

After some time, the command returns, telling you that the cluster has been created successfully.

Connect To The Cluster

After creating the cluster, we want to connect to it via kubectl. Therefore we need the configuration of the cluster, which we have to copy by scp or pscp (from Putty) from the master node. To do this, execute this command (I’m on Windows, so I use pscp):

The important part of the deployment file is the node selection for the pods/containers. In lines 25/26, the template defines to select Linux-based nodes to deploy the nginx containers:

nodeSelector:
beta.kubernetes.io/os: linux

The same is done for the IIS deployment (lines 52/53):

nodeSelector:
beta.kubernetes.io/os: windows

These properties tell the Kubernetes cluster to create the nginx pods only on Linux and the IIS pods on Windows nodes, magic!

After some time (give it a few “seconds”, the image of the IIS pods is about 5 GB!), you will be able to query the running pods and the services:

Et voilà, we are able to connect to the services via a browser…

Linux based containerWindows based container

That’s what we wanted to achieve…Linux and Windows containers managed by the same cluster! 🙂

Wrap Up

I hope you could see, how easy it is to deploy a Kubernetes cluster to Azure. In this example, I only showed how to create a hybrid cluster, to be able to run Linux and Windows based containers/pods in parallel. Nevertheless, on the ACS Engine Github repository (where this sample is basically from) you can find many other examples, e.g. to place the cluster in an existing VNET, attach disks to nodes, create a Swarm cluster etc. Check it out, play with ACS…and have fun! 🙂

4 thoughts

when I run “az deploy….” in order to deploy all the resources via the ARM template in azure,I get an error like this “The template deployment ‘azuredeploy’ is not valid according to the validation procedure. The tracking id is ’05ae1c60-b2
df-43b7-99f3-906a60bbba43′. See inner errors for details. Please see https://aka.ms/arm-deploy for usage details.”

You mean, you get an error when you want to deploy the generated template (via acs-engine)? Did you adjust the template manually? If not, try to get the latest version of acs-engine, recreate the template and check, that you have entered valid parameters in the json files you use with acs-engine. If the error persists, try and get the latest version of the template from https://github.com/Azure/acs-engine/tree/master/examples/windows.