In this post, I want to write about my experience testing and using Azure Kubernetes service to deploy a Jenkins Instance solution that is highly available and resilient. With the Kubernetes persistent volume feature, an Azure disk can be dynamically provisioned and attached to a Jenkins Instance container deployment. In another scenario, an existing Azure Disk containing data related to a software team’s Jenkins projects could be attached to a new Jenkins deployment in a kubernetes cluster, to maintain continuity. Also, data loss in the event of a Pod failure can be averted since the persistent volume storage has a lifecycle independent of any individual pod that uses the persistent volume and can be managed as a separate cluster resource.

Jenkins automation server is one of the most in demand DevOps tools today. Therefore, deploying a Jenkins instance for software development and engineering teams in a fast, flexible and highly available way is key to maintaining an efficient and smooth running CI/CD and testing process.With Azure Kubernetes, we are able to deploy multiple Jenkins instances customized for each team on a centrally controlled cluster.

Prerequisites.

All the Kubernetes related work in this post was done on my Windows 10 machine. The tools are available for both Windows and Linux.

The following tools need to be installed and configured on the working machine before creating and configuring an Azure Kubernetes cluster.
1) Azure CLI : Azure command line tool for creating Kubernetes Clusters.
2) Kubectl : Follow the link to download the Kubectl executable and put kubectl.exe somewhere in your system PATH.
3) Azure Subscription.

I’ll also mention that I used the awesome Kompose.exe tool to convert my existing Docker-Compose files to Kubernetes compatible yaml files. Obviously, the yaml files needed to be edited after the convert process.

Login to Azure and create a resource group for the Kubernetes services.

In this and following sections, I will detail the steps I followed to setup and deploy a highly available Jenkins deployment in Azure Kubernetes using first a) A dynamic Azure Disk and b) A static existing Azure Disk with data.

At this time, using PowerShell to create an Azure kubernetes cluster is not supported. I suspect this will change in the future. For now, I will use Azure CLI az aks command to create a new cluster in the new resource group:

The one line of az aks create code provisions a Kubernetes service object in the rgaks resource group. In addition, it also automatically creates another resource group containing a two node cluster with all corresponding resources as shown in the following screen shot :

After creating the Kubernetes cluster, I’ll connect to the cluster from my powershell console by running the az aks get-credentials command to set the current security context on the /.kube/config file and use the kubectl command to verify the state of the kubernetes cluster nodes:

az aks get-credentials --name akscluster0 --resource-group rgaks

kubectl.exe get nodes

Setup Persistent Storage.

At this point, the cluster nodes are ready to host a Jenkins Container Instance in a kubernetes Pod.Azure Kubernetes clusters are created with two default storage classes as displayed in the screen shot below. Both of these classes are configured to work with Azure disks.A storage class is used to define how a unit of storage is dynamically created. This saves me the step of having to create a storage class manifest file to be used by my persistent volume claim yaml file. The default storage class provisions a standard Azure disk. The managed-premium storage class provisions a premium Azure disk. I verified this using the kubectl get storageclasses command:

I selected the managed-premium class for my persistent volume claim configuration. To provision persistent storage to be used for this deployment, I created a yaml file of kind: PersistentVolumeClaim to request a storage unit of capacity 5Gi based on the managed-premium storage class.The pvc will be referenced in my Jenkins deployment yaml file to attach/map the automatically created Azure Disk volume to the Jenkins data home folder path. The following yaml file defines the dynamic creation of a 5Gi unit of storage:

I merged the persistent volume claim yaml file with the Jenkins application deployment and service yaml files to automatically provision the storage, provision the Jenkins deployment, map the persistent storage as a volume to the Jenkins instance and create a service type LoadBalancer to enable me access the Jenkins instance from outside the kubernetes cluster virtual network. The full and correctly indented yaml file can be found here on my GitHub page. Using the kubectl.exe tool, I can run a one line script that provisions the deployment, services and persistent volumes from a single yaml file:

The screen shot displays the one line kubectl script and the deployment, services and persistent volumes created:

kubectl.exe apply -f .\jenkinsbox-k8s-all-in-one.yaml

Check the status to the deployments,services and pods.

Start initial configuration of the Jenkins Instance in kubernetes container.

Use the following command to retrieve the initialAdmin default password for Jenkins:

kubectl.exe logs jenkinsbox-55f58fcbcb-2ltqx

Use the public ip address from the kubernetes service in the above screen shot to access the Jenkins initial setup page:

After initial configuration of Jenkins, I created a sample pipeline job:

Simulate failure and recovery.

To verify failure and recovery, I’ll delete the pod using the following command in the screen shot:

kubectl.exe delete pods --all

The screen shot indicates the pod deletion and immediate automatic creation of a new pod with same storage volume to match the number of replicas defined the deployment yaml file. In the next screen shot, I simply login to the Jenkins instance without going through the setup wizard of a new instance. I can also confirm that the pipeline job created in the preceding steps is still available:

Verify recovery by deleting the current Jenkins deployment and using existing Azure Disk for static disk persistent volume on new deployment.

In this example, I delete the deployment and services created in th epreceding steps and develop a yaml file config to deploy a new Jenkins instance using the existing disk provisioned above for an Azure static disk persistent storage volume mapped to a new Jenkins deployment.

Create a new deployment using the new yaml file:

kubectl.exe apply -f .\jenkinsbox-deployment-static-disk.yaml

The following screen shot confirms that all the deployment components and pods have been successfully provisioned with the existing persistent storage volume mapped to the new pod.

As soon as the new deployment and pod are running. I can login to the existing jenkins instance without a setup prompt (since the static disk contains data from the initial deployments in the preceding steps) and confirm the existing pipeline Job which is available since the existing disk is mapped to the new kubernetes pod.

My new yaml file does not have a persistent volume claim section. I simply reference the diskUri of the existing managed disk in the deployment section.I used my jenkinsbox-deployment-static-disk.yaml file on Github for the new deployment to map an existing azure disk to the new deployment. The following snippet displays the section of the yaml file that maps the existing Azure disk:

A problem I experienced during initial deployment was a failed container. After digging into the logs for the container by using the kubectl.exe get pods, kubectl.exe logs pods and kubectl.exe describe pods commands, I noticed the pod was stuck in a “ContainerCreating” loop. I also observed the following kubernetes event log messages: “MountVolume.SetUp failed for volume” and “do not have required permission”.

The pod could not attach the disk volume. This is similar to an issue that occurs using bind mounts in Docker. Resolving the issue in Docker is easy. But I couldn’t immediately determine how to resolve it within a kubernetes deployment environmet. This issue occurs because by default, non-root users do not have write permission on the volume mount path for NFS-backed storage. Some common app images, such as Jenkins and Nexus3, specify a non-root user that owns the mount path in the Dockerfile. When a container is created from this Dockerfile, the creation of the container fails due to insufficient permissions of the non-root user on the mount path.

After some research, I found the following solution that uses an InitContainer in my deployment, to give a non-root user that is specified in my Dockerfile write permissions for the volume mount path inside the container.

The init container creates the volume mount path inside the container, changes the mount path to be owned by the correct (non-root) user, and closes. Then, my Jenkins container starts with the non-root user that must write to the mount path. Because the path is already owned by the non-root user, writing to the mount path is successful. The full example on using the InitContainer can also be found here. The following is a section of my deployment yaml file that defines the InitContainer:

Having tested both Azure Disks and Azure File Share as Persistent Volume options for deploying Jenkins container in AKS and I have observed that without a doubt the Azure Disk option provides much faster performance and response times for the application.

The drawback for me though lies in the fact that the access mode for Azure Disk persistent volumes is ReadWriteOnce. This means that an Azure disk can be attached to only one cluster node at a time. In the event of a node failure or update, it could take anywhere between 1-5 minutes for the Azure disk to get detached and attached to the next available node. This means that a Pod/Container/Application may incur a short outage while Kubernetes schedules it for another node. This scenario could slightly impact the high availability offering of AKS. Hopefully, Azure will improve on this behavior soon.

Introduction.
Microsoft just updated the ASWPowerShell module to better enable Cloud administrators manage and provision cloud resources in the AWS cloud space while using the same familiar PowerShell tool. As at last count today, the AWSPowerShell module contains almost four thousand cmdlets:

This means Microsoft is committed to expanding on PowerShell functionality as a robust tool for managing both Azure and Amazon cloud platforms.
In this post I want to quickly demonstrate how to provision an AWS EC2 instance using PowerShell. The following steps help accomplish this objective.

Install the ASWPowerShell Module.
For this post, I’ll be using the version 5.1.16299.98 of Windows PowerShell as indicated in the following screen shot:

Configure AWS Credential Profile.
During initial signup for an AWS account, a root account is created with full administrative access. According to AWS best practices, while making API calls and using PowerShell to programmatically access and manage resources, a sub user account should be created with corresponding access key ID and secret key credentials. This way, if the keys are compromised, the associated user can be disabled instead of risking the compromise of the root account and all the resources associated with it.

Use the Users tab of the IAM (Identity and Access Management) console in the AWS portal to create a subuser and generate the corresponding access key ID and secret key.

After generating the keys, I’ll use the Set-AWSCredential cmdlet to save and persist the the credential keys to my local AWS SDK store for use across multiple PowerShell sessions. The Initialize-AWSDefaultConfiguration cmdlet sets the new profile and region as active within the PowerShell session. The following script accomplishes this task. Please note that the AccessKey and SecretKey parameter values are represented by variables:

Create an EC2 Key Pair.
Use the New-EC2KeyPair cmdlet to create an EC2 key pair. This cmdlet calls the Amazon Elastic Compute Cloud CreateKeyPair API. It creates a 2048-bit RSA key pair with the specified name. Amazon EC2 stores the public key and displays the private key to be saved to a file. The private key is returned as an unencrypted PEM encoded PKCS#1 private key. The private key is used during the logon operation to a virtual machine to create a password for login. If a key with the specified name already exists, Amazon EC2 returns an error.Up to five thousand key pairs can be created per region. The key pair is available only in the region in which it is created. In the following script, I create the key, assign the key pair object to a variable and save the key material property of the key pair object locally to a file:

Provision a Non-Default Virtual Private Cloud (VPC).
The first time I created my AWS account, a default VPC provisioned with a private ip address scheme.For the purpose of this post, I would prefer to create a custom non-default vpc with an address range of my choice. Unlike the default vpc, the non-default vpc does not have internet connectivity. Some extra configuration is needed to enable internet connectivity to the non-default vpc.
The following tasks are accomplished by the PowerShell script to enable internet connectivity for the custom non-default vpc:Create the non-default vpc and enable dns hostnames
Tag the vpc with a friendly name
Create a custom subnet for the vpc and tag it
Create an internet gateway and attach it to the custom vpc
Create a custom route table for internet access and associate it with the custom subnet
#Create non default virtual private cloud/virtual network, enable dns hostnames and tag the vpc resource
$Ec2Vpc = New-EC2Vpc -CidrBlock "10.0.0.0/16" -InstanceTenancy default
Edit-EC2VpcAttribute -VpcId $Ec2Vpc.VpcId -EnableDnsHostnames $true
$Tag = New-Object Amazon.EC2.Model.Tag
$Tag.Key = "Name"
$Tag.Value = "MyVPC"
New-EC2Tag -Resource $Ec2Vpc.VpcId -Tag $Tag

Logon to the EC2Instance using Remote Desktop protocol.
Login to the EC2 Instance Virtual machine can be initiated using the AWS EC2 Dashboard.The private key portion of the keypair will be used to create a password to login to the Virtual Machine as indicated in the following screen shots:

Select the EC2 Instance and click on the Connect button. On the Connect to your Instance page, click on the Get Password button.

On the Get Password page, copy and paste the private key from the keypair file into the content field and click to decrypt the key.

Copy the displayed password, download the RDP file and login to the EC2Instance. It is recommended to change the password and create a new local user after logon.

Summary:
A new class of security vulnerabilities referred to as “Speculative execution side-channel attacks” also known as “Meltdown and Spectre” were publicly disclosed by Cyber security researchers this week. Given the gravity of these flaws, many concerns have been rightly raised. In this article I will cover their impact as well as the Microsoft Cloud’s response and how they are dealing with it. I will also cover how you can mitigate and prevent the issues for on-premises environments.

General Overview:
These security flaws exploit critical vulnerabilities in modern processors whether Intel, Apple, AMD or ARM chips. They allow programs to steal data while it’s being processed on the computer. While programs are typically not permitted to read data from other programs, a malicious program can exploit Meltdown and Spectre to get hold of secrets stored in the CPU’s memory of other running programs. This might include your passwords stored in a password manager or browser, your personal photos, emails, instant messages and even business-critical documents.

Overview of Spectre Vulnerability:
Spectre breaks the most fundamental isolation between user processes and the operating system. This attack allows a program to access the CPU’s memory, and thus also the secrets, of other programs and the operating system.If your computer has a vulnerable processor’s (Intel, Apple, ARM, AMD) and runs an unpatched operating system / firmware, it is not safe to work with sensitive information without the chance of leaking the information. This applies both to personal computers as well as cloud infrastructure.

Overview of the Meltdown Vulnerability:
Meltdown breaks the isolation between different applications using a flaw in the Kernel on machines with vulnerable processor’s (Intel, Apple). It allows an attacker to trick error-free programs, which follow best practices, into leaking their secrets. In fact, the safety checks of said best practices actually increase the attack surface and may make applications more susceptible.

How to Enable Protection for On-premise and Cloud Environments:
Microsoft has provided instructions for recommended actions that enable protection of servers against these vulnerabilities. Some of these are detailed in the following link: Microsoft Protections Against Meltdown and Spectre

Impact to Enterprise Cloud Services:
Microsoft has provided updates of the impact of these flaws on enterprise cloud services in the following link: Impact to Enterprise Cloud Services

Amazon Web Services Linux Protection Recommendations:
All instances across the Amazon EC2 fleet are protected from all known threat vectors from the CVEs previously listed. Customers’ instances are protected against these threats from other instances. We have not observed meaningful performance impact for the overwhelming majority of EC2 workloads. The following link provides further recommendations and updates.Recommended Customer Actions for AWS

Conclusion:
It must be added that Public Cloud providers like Microsoft Azure and Amazon AWS are well ahead of the curve when it comes to security. Cloud providers got on top of this way before most of the industry.Public Cloud providers like Amazon AWS and Microsoft Azure are also much safer than on-premise sites considering that they employ a large and formidable army of security professionals and resources.

Personal computer users are also strongly advised to immediately download and install patches and updates available for their operating systems.

Azure Site Recovery service enables businesses ensure business continuity by keeping their workloads and applications running on Virtual Machines and physical servers available if a site goes down. Site Recovery replicates these workloads running on VMs and physical servers so that they remain available in a secondary location if the primary site isn’t available. It recovers workloads to the primary site when it’s up and running again.
In this post, I want to share a PowerShell solution I developed with the Azure Resource Manager model, to automate the deployment of Site Recovery to configure and orchestrate Windows Server 2016 Hyper-V host and guest virtual machine replication and protection to Azure Cloud.
The solution consists of a PowerShell function and module that successfully deploys and configures Azure Site Recovery for a Hyper-V to Azure replication scenario without VMM :

1) The New-HyperVASRDeployment function handles the configuration and orchestration of the Hyper-V and workloads to Azure.

2) The Install-ASRProvider module is called within the first function to execute the download, extraction of the “.\AzureSiteRecoveryProvider.exe” file, and installation of the ASR provider and Recovery services agent on the local Hyper-V host (a windows server 2016 server).

The New-HyperVASRDeployment function can be executed as a Runbook or run locally on the Hyper-V host within an elevated PowerShell console. It authenticates to azure using an Azure RunAsAccount. An Azure automation account will have to be provisioned.

Prerequisites:

1) The Install-ASRProvider.psm1 module can be downloaded from my GitHub repository. It must be copied to the PowerShell modules folder on the local host machine. If the solution is deployed as a runbook, this module will need to be imported into the Automation account module folder.
2) The Install-ASRProvider.psm1 module downloads the “.\AzureSiteRecoveryProvider.exe” file from https://aka.ms/downloaddra (Microsoft could change this link at anytime) to a local folder on the Hyper-V host that matches the $Path variable value specified in the Install-ASRProvider.psm1 module.
3) Ensure internet access is available on the Hyper-V Host.
4) An Azure automation account will have to be provisioned. This function was developed to run as a Runbook or locally. It authenticates to azure using an Azure RunAsAccount.

Overview

The New-HyperVASRDeployment function (also available at my GitHub Repo) registers the Site Recovery and RecoveryServices providers for the subscription and creates a new resource group for the site recovery resources and components.
It goes on to provision the Recovery vault, storage account, Hyper-V site container and sets the recovery services context for the subscription.
The next section makes a call to the Install-ASRProvider module that downloads, installs the ASR provider and agent, then registers the on-premise Hyper-V host with the Azure recovery services vault.
The remaining sections of the code create a replication policy, map the policy to the existing Hyper-V site, enables ASR protection on the selected Hyper-V host virtual machine workload and initiates synchronization with the Azure recovery vault.
The end result is that the virtual machine shows up in the Replicated Items blade of the Recovery Services vault as protected and 100% synchronized in Azure as shown in the screen shots below.
In the future, I hope to share a function that automates the Site Recovery failover of a protected virtual machine.

The following screen shots show the solution running within a PowerShell console:

1)

2)

3)

4)

The next set of screen shots display the registered server(Hyper-V host) in Azure, the VM Replication state in Hyper-V manager on-premise and the replication item in Azure after the successful execution of the functions:

Replication Item after successful synchronization of the Virtual machine:

The following Install-ASRProvider.psm1 module is called by the New-HyperVASRDeployment above. It downloads and extracts the ASR Provider file, then installs the ASR provider and Recovery Vault Agent and registers the Server in the Azure Recovery Services Vault.

PowerShell helps me better understand the internal workings of complex Cloud based operations that in many instances have a lot of moving parts. Better understanding helps determine ways to automate these operations that could otherwise take much time and repetition to accomplish.I developed the following Deploy-ManagedEncryptedLinuxVM PowerShell function to automate the provisioning, installation of LAMP Stack software components and encryption of one or more Managed Disk Azure VM(s) running CentOS 7.3. using Azure Disk Encryption. The function is also available at my GitHub Repository .

a) An Azure Key Vault needs to be created and enabled for Template Deployment and DiskEncryption. The Key Vault is a hardened container for the keys and secrets generated during the encryption process. The Key Vault must be in the same region as the VMs to be encrypted.

b) Additionally, an Azure Active Directory (AAD) application is required to write secrets to the Key Vault. After the Application is created, authorize the application to use the key or secret in the Key Vault by using the Set-AzureRmKeyVaultAccessPolicy cmdlet. Microsoft provides an Azure Disk Encryption prerequisite PowerShell script to help create an Azure AD application, create a key vault, or set up an existing key vault for encryption.

Overview

The Deploy-ManagedEncryptedLinuxVM Function takes the Location, SubscriptionName and ResourceGroupName variables as parameters. The first section of the function uses the Azure Automation RunAs account to successfully authenticate against the Azure Subscription. The next section declares and initializes variables related to the Key Vault, Application ID and Template Uris for deploying the Virtual Machine templates.

The Check Key Vault Access Policy region of the function confirms that the Azure Application has been granted appropriate permissions to the existing key vault. We then deploy the ARM templates (available at my GitHub repository) to provision the Virtual machines. The template parameters file is available here. The ARM template also runs a bash script that installs the LAMP (Linux, Apache, MariaDB and Python) Stack to the CentOS Virtual Machine. The final section of the function initiates disk encryption and restarts the VM to complete the process.

The final section of the function sends an email notification to the subscription owner(s) with details of the VM(s) encryption status for review.

Azure Storage Files is a sub service of Azure Storage Accounts. It offers fully managed file shares in the cloud that are accessible via the industry standard Server Message Block (SMB) protocol (also known as Common Internet File System or CIFS). The cloud File shares can be mounted concurrently by cloud or on-premises deployments of Windows, Linux, and macOS machines.

At a recent cloud migration engagement, a key requirement was the automation of Azure storage file sync/copy to a secondary storage account in the same Azure subscription to help mitigate against accidental file deletion.

As of this writing, Azure Backup and snapshots are not supported for Azure File Storage. Microsoft recommends using Robocopy, the AzCopy CLI tool or a 3rd party backup tool to copy or backup Azure File storage data. Robocopy needs a UNC path,which will require the Azure Storage File Share be mapped as a local network drive for local access to the files. This option did not meet the requirement for automation. The AzCopy tool, does not require local access to a mapped drive, but needs to be installed on the Azure VM or on-premise server.

The ideal automated solution should be deployed in Azure and in addition, trigger email notifications with details of the copied file storage items after processing the files. I developed the following PowerShell function to meet these requirements. Hopefully, Azure Backup and Snapshot support for Azure File Storage will be rolled out soon. A feature request for File Storage Account backups or snapshots currently exists at this link.

This function (which can also be downloaded at my GitHub Repository) loops through the existing file shares and directories, then copies the existing files based on the last modified time stamp property being greater than one hour. I deployed this solution as a Time triggered PowerShell Azure Function App executed once at the top of every hour.

At this time, I don’t think Azure Function Apps support Azure Key Vault for secure authentication to Azure( I could be wrong though and stand corrected). For this Function App to authenticate with Azure, I generated and uploaded an AES256 Encryption key file used to encrypt/decrypt the password in order to generate an encrypted standard password file to be referenced by the function App. The following screen shot displays the Function App settings for the encrypted Azure logon password, Azure credential username, encrypted sendgridpassword and the local time zone.

One of the more relevant and cost saving tasks when working in Azure Cloud is fast and efficient shutdown and startup of virtual machines. Some of the options available could be to :

a) Iterate through a csv list of VMs or Resource Group of VMs and serially shutdown or startup the VMs. This can be accomplished using a Foreach loop. A Downside of this option is that it’s slow and could take a while depending on the number of VMs. In my lab test, the shutdown and startup 60 VMs took almost 3 hours at 3 minutes per VM for each action.

b) A better option would be to execute the shutdown/startup scripts on the VMs in parallel. Using the Foreach -Parallel in a Powershell workflow took less time, but almost 2 hours to shutdown and startup 60 VMs. Too much time. This function can only be deployed as a Runbook set to run on a schedule. While this met the requirement for automation, we also wanted a solution that could be run easily from a laptop as a function or module.

c) Of course there’s also the option of using the Web based portal to shutdown and startup 60 VMs or more. Well this is simply not a viable and scalable option.

While researching the most efficient and fastest way to accomplish this task, I came upon the Invoke-Parallel function by Cookie.Monster. I developed the Set-AzureRMVmPowerState PowerShell function to utilize Invoke-Parallel as a module in the shutdown and startup of all 60 VMs in about 30 minutes.

The Invoke-Parallel works by creating multiple runspaces to execute my Set-AzureRmVMPowerState function and concurrently shutdown or startup all the Virtual Machines in the shortest possible time. My Set-AzureRmVMPowerState function accepts four parameters : State (Start/Stop), SubscriptionName, ResourceGroupName parameter set or a FilePath for the csv file list of VMs. An email notification is sent to the subscription admin with a list of Azure VMs processed and their current power state.
The function is also available at my github repository.