At the time of writing this guide, Azure Kubernetes Service (AKS) is still in technical preview.
Some aspects of AKS features may change prior to general availability (GA).
Cloudbees team recommends reviewing the Microsoft’s Azure Kubernetes Service (AKS) documentation.

Terms and Definitions

Jenkins

Jenkins is an open-source automation server. Jenkins is an
independent open-source community, to which CloudBees actively contributes. You can find more information
about Jenkins and CloudBees contributions on the CloudBees site.

CloudBees Core

Commercial version of Jenkins based on Jenkins Long-Term Support (LTS) releases with frequent patches by CloudBees. CloudBees Core also provides a number of plugins that help organizations address main needs of enterprise installations: Security, Continuous Delivery, etc.

CloudBees Core Operations Center

operations console for
Jenkins that allows you to manage multiple Jenkins masters. See details on the CloudBees site.

Architectural Overview

This section provides a high-level architectural overview of CloudBees Core, designed to help you understand how CloudBees Core works, how it integrates with Kubernetes, its network architecture and how Managed Masters and build agents are provisioned.

CloudBees Core is essentially a set of Docker Containers that can be deployed to run a cluster of machines within the Kubernetes container management system. Customers are expected to provision and configure their Kubernetes system before installing CloudBees Core.

CloudBees Core includes the CloudBees Core Operations Center that provisions and manages CloudBees Managed Masters and Team Masters.
CloudBees Core also enables Managed Masters and Team Masters to perform dynamic provisioning of build agents via Kubernetes.

Machines and Roles

CloudBees Core is designed to run in a Kubernetes cluster.
For the purposes of this section, you need to know that a Kubernetes cluster is a set of machines (virtual or bare-metal), which run Kubernetes.
Some of these machines provide the Kubernetes control plane as Kubernetes Masters.
They control the Containers that run on the other type of machines known as Kubernetes Nodes. The CloudBees Core containers will run on the Kubernetes Nodes.

The Kubernetes Masters provide an HTTP-based API that can be used to manage the cluster, configure it, deploy containers, etc.
A command-line client called kubectl can be used to interact with Kubernetes via this API.
You should refer to the Kubernetes documentation for more information on Kubernetes.

CloudBees Core Docker Containers

The Docker containers in CloudBees Core are:

cloudbees-cloud-core-oc: CloudBees Core Operations Center

cloudbees-core-mm: CloudBees Core Managed Master

The Docker containers used as Jenkins build agents are specified on a per Pipeline basis and are not included in CloudBees Core.
Refer to the example Pipeline in the Agent Provisioning section for more details.

The cloudbees-cloud-core-oc, cloudbees-core-mm and build agent container images can be pulled from the public Docker Hub repository or from a private Docker Registry that you deploy and manage.
If you wish to use a private registry, you will have to configure your Kubernetes cluster to do that.

CloudBees Core Kubernetes Resources

CloudBees Core is deployed to a Kubernetes cluster by applying the CloudBees Core Kubernetes configuration file, cloudbees-core.yml a YAML file that defines the set of Kubernetes resources required by CloudBees Core.
This is done via a single command-line: kubectl create -f cloudbees-core.yml but to understand what happens when that command is executed, you need to understand several more Kubernetes concepts:

Kubernetes concepts

Pod

A set of Containers that share storage volumes and a network interface.

ServiceAccount

Defines an account for accessing the Kubernetes API.

Role

Defines a set of permission rules for access to the Kubernetes APIs.

RoleBinding

Binds a ServiceAccount to a Role.

ConfigMap

A directory of configuraton files made available on all Kubernetes Nodes.

StatefulSet

Managing deployment and scaling of a set of Pods.

Service

Provides access to a set of Pods at one or more TCP ports.

Ingress

Uses hostname and path of an incoming request to map the request to a specific Service.

When you initially deploy CloudBees Core, you will see a Operations Center Pod running and within that, a Operations Center Container. You can login to Operations Center and provision Managed Masters to run your Pipelines.

Visualizing CloudBees Core Architecture

The diagram below illustrates the CloudBees Core architecture on Kubernetes.
The diagram shows three Kubernetes Master Nodes, which are the three dotted-line and overlapping rectangles on the left.
The diagram also shows two Kubernetes Worker Nodes, which are the two dotted-line large rectangles in the center and right.

Here’s the key for the colors used in the diagram:

Green: processes which are part of Kubernetes

Pink: Kubernetes resources created by installing and running CloudBees Core

Yellow: Kubernetes resources required by CloudBees Core

Figure 1. CloudBees Core Architecture

Kubernetes Master

Running on each Kubernetes Master node, there are the Kubernetes processes that manage the cluster: the API Server, the Controller Manager and the Scheduler.
In the bottom left of the diagram are resources that are created as part of the CloudBees Core installation, but that are not really tied to any one node in the system.

Kubernetes Nodes

On the Kubernetes Nodes and shown in green above is the kubelet process, which is part of Kubernetes and is responsible for communicating with the Kubernetes API Server and starting and stopping Kubernetes Pods on the Node.

On one Node, you see the Operations Center Pod which includes a Master Provisioning plugin that is responsible for starting new Master Pods.
On the other node you see a Master Pod, which includes the Jenkins Kubernetes Plugin and uses that plugin to manage Jenkins Build Agents.

Each Operations Center and Master Pod has a Kubernetes Persistent Volume Claim where it stores its Jenkins Home directory.
Each Persistent Volume Claim is backed by some form of storage service, such as an EBS volume on AWS or an NFS drive in an OpenShift environment. When a Master Pod is moved to a new Node, its storage volume must be detached from its old Node and then attached to the Pod’s new node.

Master Provisioning

One of the benefits of CloudBees Core is the easy provisioning of new Jenkins Managed masters from the Operations Center web interface. This feature is provided by the CloudBees Core Master Provisioning Plugin for Jenkins. When you provision a new master, you must specify the amount of memory and CPU to be allocated to the new master, and the provisioning plugin will call upon the Kubernetes API to create a master.

The diagram below shows what happens when a new Master is launched via Operations Center. First, CJOC’s Master Provisioning Kubernetes Plugin calls Kubernetes to provision a new StatefulSet to run the Managed Master Pod.

Figure 2. Master Provisioning

Agent Provisioning

Agents are created and destroyed in CloudBees Core by the Jenkins Kubernetes Plugin.
A Jenkins Pipeline can specify the build agent using the standard Pipeline syntax.
For example, below is a CloudBees Core Pipeline that builds and tests a Java project from a GitHub repository using a Maven and Java 8 Docker image:

In the above example, the build agent container image is maven:3.5.2-jdk-8-alpine.
It will be pulled from the Docker Registry configured for the Kubernetes cluster.

The diagram below shows how build agent provisioning works.
First, when the Pipeline runs, the Kubernetes Plugin on the Managed Master calls Kubernetes to provision a new pod to run the build agent container.
Second, Kubernetes launches the build agent pod to execute the Pipeline.

Figure 3. Agent Provisioning

CloudBees Core Required Ports

CloudBees Core requires open ports:

80 for http access to the web interface of Operations Center and Managed Masters

443 for https access to the web interface of Operations Center and Managed Masters

Network Encryption

Network Communication between Kubernetes clients such as kubectl, Kubernetes masters and nodes is encrypted via TLS protocol.
The Kubernetes Managing TLS in a Cluster document explains how Certificates are obtained and managed by a cluster.

Communication between application containers running on a Kubernetes cluster, such as Operations Center and Managed Masters, can be encrypted as well but this requires the deployment of a Network Overlay technology such as Weave Works.
End-to-end Web Browser to CloudBees Core communications can be TLS encrypted by configuring the Kubernetes Ingress that provides access to CloudBees Core to be the termination point for SSL.
Network Overlay and SSL termination configuration is covered in a separate section.

High Availability

Kubernetes can be configured for high availability by using at least three Kubernetes Masters on three separate machines in different Availability Zones.

Persistence

Operations Center and Managed Masters store their data in file-system directory, known as Jenkins Home. Operations Center has its own Jenkins Home, and each Master also has one.

Azure Resource Groups

Microsoft’s Azure Kubernetes Service (AKS) reduces the complexity of Kubernetes cluster management.
Azure Kubernetes Service (AKS) automatically creates two Azure resource groups in each Kubernetes deployment.
AKS creates cluster resources in the resource group provided in the az aks create command.
It then places infrastructure resources in a secondary resource group.

AKS creates the secondary resource group to make it easier to clean up the Kubernetes cluster.
Removing infrastructure resources from the default resource group prevents users from accidentally deleting an Azure infrastructure resource the new Azure Kubernetes Service (AKS) cluster requires.
Hiding the supporting infrastructure details of Kubernetes cluster also allows the CloudBees Core administrator to focus on CloudBees Core administration, not Azure Kubernetes Service (AKS) infrastructure.

Cluster Sizing and Scaling

This document provides general recommendations about sizing and scaling a Kubernetes cluster for CloudBees Core starting with some general notes about minimum requirements and ending with a table of more concrete sizing guidelines recommended by CloudBees.

General notes

When sizing and scaling a cluster you should consider the operational characteristics of Jenkins. The relevant ones are:

Jenkins Masters are memory and disk IOPS bound, with some CPU requirements as well. Low IOPS results into longer startup times and worse general performance. Low memory results into slow response time.

Build Agents requirements depend on the kind of tasks being executed on them.

Pods are defined by their CPU and memory requirement and they can’t be split across multiple hosts.

It is recommended to use hosts that are big enough so that they can host several pods (Rule of thumb : 3-5 pods per host) at the same time to maximize their actual use.

Example: You are running builds requiring 2 GB of memory each. You need configure pods to have 2 GB each for supporting such builds. The rule of thumb says you should have hosts with 6-10 GB of memory (3 x 2 - 5 x 2).

Depending on your cloud provider, it may be possible to enable auto-scaling in Kubernetes to match with the actual requirements and reduce the operational costs.

If you don’t have auto-scaling in your environment, we recommend you to plan extra capacity in order to sustain hardware failure.

Storage

Each Managed Master is provisioned on a separate Persistent Volume (PV). It is recommended to use a storage class with the most IOPS available.

The host storage is not getting used by Managed Masters but depending on the instance type you may have restrictions on the kind of block storage you can use (for example, on Azure, you need to use an instance type ending with s).

Disk space on the hosts is necessary to host docker images, containers and volumes. Build workspaces will be on host storage so there must be enough free disk space available on nodes.

CPU

By default, a Managed Master requires 1 CPU. Each build agent also requires CPU, so what will determine the total CPU requirement is :

(mostly static) The number of Managed Masters multiplied by the number of CPU each of them requires.

(dynamic) The number of concurrent build agents used by the cluster multiplied by the CPU requirement of pod template. A minimum amount of 1 CPU is recommended for a pod template but you can use more cpus if parallel processing is required by the task.

Most build tasks are CPU-bound (compilation, test executions). So it is quite important when defining pod templates not to underestimate the number of cpus to allocate if you want good performance.

Memory

By default, a Managed Master requires 3 GB of RAM.

To determine the total memory requirement, take into account:

(mostly static) The number of Managed Masters multiplied by the amount of RAM each of them requires.

(dynamic) The number of concurrent build agents used by the cluster multiplied by the memory requirement of pod template

Memory also impacts performance. Not giving enough memory to a Managed Master will cause additional garbage collection and reduced performance.

Master Sizing Guidelines

Below are some more concrete sizing guidelines compiled by CloudBees Support Engineers:

Table 1. Master sizing guidelines

Requirement

Baseline

Rationale

Team Size

10

Most agile resources warn against going above 10 team members. Keeping the team size at 10 or below facilitates the sharing of knowledge about Jenkins and pipeline best practices.Three items

Average Weekly Users

20

Besides the team themselves, other non-team collaborators often must access the team’s Jenkins to download artifacts or otherwise collaborate with the team. This includes API clients.

Serving the Jenkins user interface impacts IO and CPU consumption and will also result in increased memory usage due to the caching of build results.

CPU Cores

4

A Jenkins of this size should have at least 4 CPU cores available.

Maximum Concurrent Builds

50

Healthy agile teams push changes multiple times per day and may have a large test suite including unit, integration and automated system tests.

We generally observe Jenkins easily handles up to 50 simultaneous builds, with some Jenkins regularly running many multiples of this number. However, poorly written or complicated pipeline code can significantly affect the performance and scalability of Jenkins since the pipeline script is compiled and executed on the master.

To increase the scalability and throughput of your Jenkins master, we recommend that Pipeline scripts and libraries be as short and simple as possible. This is the number one mistake teams make. If build logic can possibly be done in a Bash script, Makefile or other project artifact, Jenkins will be more scalable and reliable. Changes to such artifacts are also easier to test than changes to the Pipeline script

Maximum Number of Pipelines
(Multi-branch projects)

75

Well-designed systems are often composed of many individual components. The microservices architecture accelerates this trend, as does the maintenance of legacy modules.

Each pipeline can have multiple branches, each with its own build history. If your team has a high number of pipeline jobs, you should consider splitting your Jenkins further.

Recommended Java Heap Size

4 GB

We regularly see Jenkins of this size performing well with 4 gigabytes of heap. This means setting the -Xmx4g as recommended in option B of our KB article on the topic.

If you observe that your Jenkins instance requires more than 8 gigabytes of heap, your Jenkins likely needs to be split further. Such high usage could be due to buggy pipelines or perhaps non-verified plugins your teams may be using.

Important

At the time of this writing, Azure Kubernetes Service (AKS) does not support dynamic auto-scaling.
It is also not on the roadmap until after the Azure Kubernetes Service (AKS) GA.

Setting up a Private and Encrypted Network

Use a Private and Encrypted Network to ensure that all network communication between the CloudBees Core Operations Center, Managed Masters and Build Agents is encrypted.

Network communication between the Kubernetes components running on Kubernetes Masters and Kubernetes nodes is encrypted using the TLS protocol. Each Kubernetes cluster includes a Certificate Authority (CA) that it uses to create and validate the TLS certificates used when Kubernetes components communicate with the Kubernetes API server. Refer to the Kubernetes documentation page Manage TLS Certificates in a Cluster for more information.

Applications deployed to a Kubernetes cluster, such as CloudBees Core, are not protected by Kubernetes built-in TLS encryption. To ensure that all network communication between application components is encrypted, configure your Kubernetes cluster to use a Network Policy Provider that supports encryption. The Kubernetes documentation Declare Network Policy page lists the providers that are supported.

Select a Network Policy Provider: Weave Net

The CloudBees Core development team has had some experiences using Weave Net with Kubernetes, but has not deployed Weave Net in production, so Weave Net is only our preliminary recommendation. We selected Weave Net for this document because of those experiences and because Weave Net uses IPSec protocol and its encryption uses the well-known NaCl library.

Weave Net is open source software licensed under the Apache Software Licence v2 and developed by Weaveworks with source code hosted at the Weave GitHub project. You can read more about the product and its requirements on the Weave Net overview page.

The rest of this section gives an outline of the steps you need to take to configure Weave Net to use encryption. In summary, if you provide a password-secret to Weave Net then encryption is enabled. The full documentation for this is on the Weave Net documentation page Changing Configuration Options.

Create a Kubernetes Secret to be used by Weave Net

First you need to create a Kubernetes Secret containing the password you wish to use for Weave Net. For example, you can do this via Bash. Create a weave-secret file with your password:

Ingress TLS Termination

Ingress TLS Termination should be used to ensure that network communication to the CloudBees Core web interfaces is encrypted from end to end.

If you would like to ensure that web browser to CloudBees Core communications is encrypted end-to-end then you will need to change the Kubernetes Ingress used by CloudBees Core to use your TLS Certificates, thereby making it the termination point for TLS.

Store your TLS Certs in a Kubernetes Secret

To make your TLS Certs available to Kubernetes, you must create a Kubernetes Secret using the Kubernetes command-line tool kubectl. For example, if your certs are in /etc/mycerts you would issue this command to create a secret named my-certs:

Change the two CloudBees Core Ingresses to be the TLS termination point

Next, you must configure the CloudBees Core Ingress to use your TLS Certs.
You do this by editing the Ingress resource named "cjoc" in the cloudbees-core.yml configuration file, uncommenting the lines around tls: and setting the correct hostname and setting the secretName to the secret that holds your TLS certificates.
For example, if your CloudBees Core host name is cje.example.org and your TLS Certs secrets file is named my-certs then this is what the new CloudBees Core resource should look like.

The domain name must be the same than what is used in the browser otherwise a default backend - 404 will be returned.

Configuring Persistent Storage

For persistence of CloudBees Core Operations Center and Managed Master data, CloudBees Core must be able to dynamically provision persistent storage.
When deployed, the system will provision storage for CloudBees Core Operations Center’s JENKINS_HOME directory and whenever a new Managed Master is provisioned, CloudBees Core Operations Center will provision storage for that master’s JENKINS_HOME.

On Kubernetes, dynamic provisioning of storage is accomplished by creating a Persistent Volume Claim which will use a
Storage Class to coordinate with a storage provisioner to provision that storage and make it available to CloudBees Core.

Please refer to the next section in order to set up a storage class for your environment, if applicable.

Azure Storage Class Considerations

Azure Kubernetes Service (AKS) provides each cluster with two Kubernetes Storage Classes that work Azure Managed Disk.
The default storage class use a standard Azure Managed Disk.
The other storage class is managed-premium storage class use a premium Azure Managed Disk.

Integrating Single Sign-on

Once your CloudBees Core cluster is up and running you can integrate it with a SAML-based single sign-on (SSO) system and configure Role Based Authentication Controls (RBAC).
This is done by installing the Jenkins SAML plugin, configuring it to communicate with your IDP and configuring your IDP to communicate with CloudBees Core.

Prerequisites for this task

You will need the following items before you setup SAML based SSO and RBAC:

Enable and configure SAML authentication

Click the Enable security checkbox and confirm there is a SAML 2.0 option in the Security Realm setting.
If it is not there, then the Jenkins SAML Plugin is not installed and you need to install the SAML plugin.

Enter the IDP Metadata (XML data) and specify the attribute names that your IDP uses for username, email and group membership.
When you are ready, click Save to store the new security settings.

Export Service Provider Metadata to your IDP

After you save your security settings, CloudBees Core Operations Center will report your Service Provider metadata (XML data).
You must copy this data and give it to your IDP administrator, who will add it to the IDP configuration.

You can find the Service Provider metadata by following a link on the Configure Global Security page at the end of the SAML section. The link looks like this:

Service Provider Metadata which may be required to configure your Identity
Provider (based on last saved settings).

Login to CloudBees Core Operations Center

Once your IDP administrator confirms that your IDP Metadata has been added to the IDP, attempt to login via the Login link in CloudBees Core Operations Center.

Setup RBAC

Refer to the "Role-based matrix authorization strategy" help in Manage Jenkins›Configure Global Security to enable Role Based Access Controls (RBAC).

The registered trademark Jenkins® is used pursuant to a sublicense from the Jenkins project and Software in the Public Interest, Inc. Read more at
www.cloudbees.com/jenkins/about.

Apache, Apache Ant, Apache Maven, Ant and Maven are trademarks of
The Apache Software Foundation. Used with permission. No endorsement by
The Apache Software Foundation is implied by the use of these marks.

Other names may be trademarks of their respective owners.
Many of the designations used by manufacturers and sellers
to distinguish their products are claimed as trademarks. Where those
designations appear in this book, and CloudBees was aware
of a trademark claim, the designations have been printed in caps or initial
caps.

While every precaution has been taken in the preparation of this book,
the publisher and authors assume no responsibility for errors or omissions,
or for damages resulting from the use of the information contained
herein.