Deploying and Scaling Jenkins on Kubernetes

Expert Training in Kubernetes and Rancher

Introduction

Jenkins is an open-source continuous integration and continuous delivery tool, which can be used to automate building, testing, and deploying software. It is widely considered the most popular automation server, being used by more than a million users worldwide. Some advantages of Jenkins include:

Open-source software with extensive community support

Java-based codebase, making it portable to all major platforms

A rich ecosystem of more than 1000 plugins

Jenkins works well with all popular Source Control Management systems (Git, SVN, Mercurial and CVS), popular build tools (Ant, Maven, Grunt), shell scripts and Windows batch commands, as well as testing frameworks and report generators. Jenkins plugins provide support for technologies like Docker and Kubernetes, which enable the creation and deployment of cloud-based microservice environments, both for testing as well as production deployments.

Jenkins supports the master-agent architecture (many build agents completing work scheduled by a master server) making it highly scalable. The master’s job is to schedule build jobs, distribute the jobs to agents for actual execution, monitor the agents, and get the build results. Master servers can also execute build job directly.

The agents’ task is to build the job sent by the master. A job can be configured to run on a particular type of agent, or if there are no special requirements, Jenkins can simply choose the next available agent.

Jenkins scalability provides many benefits:

Running many build plans in parallel

Automatically spinning up and removing agents to save costs

Distributing the load

Even if Jenkins includes scalability features out-of-the-box, the process of configuring scaling is not always straightforward. There are many options available to scale Jenkins, but one of the powerful options is to use Kubernetes.

What is Kubernetes?

Kubernetes is an open-source container orchestration tool. Its main purpose is to manage containerized applications on clusters of nodes by helping operators deploy, scale, update, and maintain their services, and providing mechanisms for service discovery. You can learn more about what Kubernetes is and what it can do by checking out the official documentation.

Kubernetes is one of the best tools for managing scalable, container-based workloads. Most applications, including Jenkins, can be containerized, which makes Kubernetes a very good option.

Project Goals

Before we begin, let’s take a moment and describe the system we are attempting to build.

We want to start by deploying a Jenkins master instance onto a Kubernetes cluster. We will use Jenkins’ kubernetes plugin to scale Jenkins on the cluster by provisioning dynamic agents to accommodate its current workloads. The plugin will create a Kubernetes Pod for each build by launching an agent based on a specific Docker image. When the build completes, Jenkins will remove the Pod to save resources. Agents will be launched using JNLP (Java Network Launch Protocol), so we the containers will be able to automatically connect to the Jenkins master once up and running.

Docker Hub account: We will need an account with a container image repository to push the custom images for our Jenkins master and agents.

GCP account: We will provision our Kubernetes cluster on GCP. The free-tier of Google’s cloud platform should be enough to complete this guide.

Building Custom Images for Jenkins

Let’s start by building custom images for our Jenkins components and pushing them to Docker Hub.

Log in to the Linux server where you will be running Rancher and building images. If you haven’t already done so, install Docker and Rancher on the host by following the Rancher installation quick start guide. Once the host is ready, we can prepare our Dockerfiles.

Writing the Jenkins Master Dockerfile

We can begin by creating a file called Dockerfile-jenkins-master in the current directory to define the Jenkins master image:

[root@rancher-instance jenkins-kubernetes]# docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username:
Password:
Login Succeeded

Now, push the image to your Docker Hub account:

Note: In the command below, be sure to substitute your own Docker Hub account again.

If everything was successful, you should see something like this in your Docker Hub account:

Using Rancher to Deploy a Cluster

Now that our images are published, we can use Rancher to help us deploy a GKE cluster. If you set up Rancher earlier, you should be able to log into your instance by visiting your server’s IP address with a web browser.

Next, create a new GKE cluster. You will need to log in to your Google Cloud account to create a service account with the appropriate access. Follow the Rancher documentation on creating a GKE cluster to learn how to create a service account and the provision a cluster with Rancher.

Deploying Jenkins to the Cluster

As soon as the cluster is ready, we can deploy the Jenkins master and create some services. If you are familiar with kubectl, you can achieve this from command line, but you can easily deploy all of the components you need through Rancher’s UI.

Regardless of how you choose to submit workloads to your cluster, create the following files on your local computer to define the objects you need to create.

Start by creating a file to define the Jenkins deployment:

[root@rancher-instance k8s]# vi deployment.yml

Inside, paste the following:

Note: Make sure to change <dockerhub_user> to your Docker Hub account name in the file below.

One will be a LoadBalancer service which will provision a public IP allowing us to access Jenkins from Internet. The other one will be a ClusterIP service needed for internal communication between master and agents that will be provisioned in the future:

From Rancher, click on your managed cluster (called jenkins in this demo). In the upper-left menu, select the Default project and then select the Workloads tab.

From here, click Import YAML. On the page that follows, click the Read from a file button in the upper-right corner. Choose the local deployment.yml file you created on your computer and click Import.

Rancher will deploy a pod based on your Jenkins master image to the cluster:

Next, we need to configure a way to access the UI on the Jenkins master.

In Load Balancing tab, follow same process as you did to import the previous file. Click the Import YAML button, followed by the Read from a file button. Next, select the service.yml file from your computer and click the Import button:

Rancher will begin to create your services. Provisioning the load balancer may take a few minutes.

As soon as service is marked as Active, you can find its public IP address by clicking on the three vertical dots at the right end of the load balancer’s row and select View/Edit YAML. From here, scroll down to find the IP address under status > loadBalancer > ingress > ip:

We can access the Jenkins UI by typing this IP into a web browser:

Configuring Dynamic Build Agents

With the Jenkins master up and running, we can go ahead and configure dynamic build agents to automatically spin up Pods as necessary.

Disabling the Default Master Build Agents

In the Jenkins UI, under Build Executor Status on the left side, two executors are configured by default, waiting to pick up build jobs. These are provided by the Jenkins master.

The master instance should only be in charge of scheduling build jobs, distributing the jobs to agents for execution, monitoring the agents, and getting the build results. Since we don’t want our master instance to execute builds, we will disable these.

Click on Manage Jenkins followed by Manage Nodes.

Click the gear icon associated with the master row.

On the following page, set # of executors to 0 and click Save.

The two idle executors will be removed from the Build Executor Status on the left side of the UI.

Gathering Configuration Information

We need a few pieces of information to configure Jenkins to automatically provision build agents on our Kubernetes cluster. We need three pieces of information from our GCP account and one from our ClusterIP service.

In your GCP account, select Kubernetes Engine, followed by Clusters and then click on the name of your cluster. In the Details column, copy the Endpoint IP address for later reference. This is the URL we need to give Jenkins to connect to the cluster:

Next, click Show credentials to the right of the Endpoint. Copy the Username and Password for later reference.

Now, switch over to the Rancher UI. In the upper-left menu, select the Default project on Jenkins cluster. Select the Workloads tab in the upper navigation pane and click the Service Discovery tab on the page:

Click on the three vertical dots associated with the jenkins-jnlp row and click View/Edit YAML. Copy values in the spec > clusterIP and spec > ports > port for later reference.

Configuring the Jenkins Kubernetes Plugin

Back in the main Jenkins dashboard, click on Manage Jenkins, followed by Manage Plugins:

Click the Installed tab and check that the Kubernetes plugin is installed:

We can now configure the plugin. Go to Manage Jenkins and select Configure System:

Scroll to the Cloud section at the bottom of the page. Click on Add a new cloud and select Kubernetes.

On the form that follows, in the Kubernetes URL field, enter https:// followed by the cluster endpoint IP address you copied from your GCP account.

Under Credentials, click the Add button and select Jenkins. On the form that appears, enter the username and password you copied from your GCP account and click the Add button at the bottom.

When you return to the Kubernetes form, select the credentials you just added from the Credentials drop down menu and click the Test Connection button. If the configuration is correct, the test will show “Connection test successful”.

Next, in the Jenkins tunnel field, enter the IP address and port that you retrieved from the jenkins-jnlp service in the Rancher UI, separated by a colon:

Now, scroll down to the Images section at the bottom of the page, click the Add Pod Template button, and select Kubernetes Pod Template. Fill out the Name and Labels fields with unique values to identify your first agent. We will use the label to specify which agent image should be used to run each build.

Next, in the Containers field, click the Add Container button and select Container Template. In the section that appears, fill out the following fields:

Next, click the Add Pod Template button and select Kubernetes Pod Template again. Repeat the process for the second agent image you created. Make sure to change the values to refer to your second image where applicable:

Click the Save button at the bottom to save your changes and continue.

Testing Dynamic Build Jobs

Now that our configuration is complete, we can create some build jobs to ensure that Jenkins can scale on top of Kubernetes. We will create five build jobs for each of our Jenkins agents.

On the main Jenkins page, click New Item on the left side. Enter a name for the first build of your first agent. Select Freestyle project and click the OK button.

On the next page, in the Label Expression field, type the label you set for your first Jenkins agent image. If you click out of the field, a message will appear indicating that the label is serviced by a cloud:

Scroll down to the Build Environment section and check Color ANSI Console Output.

In the Build section, click Add build step and select Execute shell. Paste the following script in the text box that appears:

Create another four jobs for the first agent by clicking New Item, filling out a new name, and using the Copy from field to copy from your first build. You can save each build without changes to duplicate the first build exactly.

Next, configure the first job for your second Jenkins agent. Click New Item, select a name for the first job for the second agent, and copy the job from your first agent again. This time, we will modify the fields on the configuration page before saving.

First, change the Label Expression field to match the label for your second agent.

Next, replace the script in the text box in the Build section with the following script:

Create four more builds for your second agent by copying from the job we just created.

Now, go to the home screen and start all the ten jobs you just created by clicking on the icon on the far right side of each row. As soon as you start them, they will be queued for execution as indicated by the Build Queue section:

After a few seconds, Pods will begin to be created to execute the builds (you can verify this in Rancher’s Workload tab). Jenkins will create one Pod for each job. As each agent is started, it connects to the master and receives a job from the queue to execute.

As soon as an agent finishes processing its job, it is automatically removed from the cluster:

To check the status of our jobs, we can click on one from each agent. Click the build from the Build History and then click Console Output. Jobs executed by the first agent should specify that the jenkins-slave1 Docker image was used, while builds executed by the second agent should indicate that the jenkins-slave2 image was used:

If you see the output above, Jenkins is configured correctly and functioning as intended. You can now begin to customize your Kubernetes-backed build system to help your team test and release software.

Conclusion

In this article, we configured Jenkins to automatically provision build agents on demand by connecting it with a Kubernetes cluster managed by Rancher. To achieve this, we completed the following steps:

Created a cluster using Rancher

Created custom Docker images for the Jenkins master and agents

Deployed the Jenkins master and an L4 LoadBalancer service to the Kubernetes cluster

The main purpose of this article was to highlight the basic configuration necessary to set up a Jenkins master and agent architecture. We saw how Jenkins launched agents using JNLP and how the containers automatically connected to the Jenkins master to receive instructions. To achieve this, we used Rancher to create the cluster, deploy a workload, and monitor the resulting Pods. Afterwards, we relied on the Jenkins Kubernetes plugin to glue together all of the different components.