It is about Jenkins

We need to prepare the hosting environments for our DevOps pipeline. This will require us to install several different components. The first core component, of course, is Jenkins. We will also install a variety of other tools in this post.

Our CI/CD Pipeline = Jenkins + Python + Supporting Software

What is Jenkins

Jenkins is the open source automation server written in Java that will build, test, and manage the overall integration and deployment pipeline. The important concept to remember is that it automates the non human part of the whole software development process, which include such things as continuous integration, continuous delivery, and more.

Jenkins has several built-in facilities to automate the CI CD pipeline.

This is the pipeline that we will automate fully with Python code as you can see below.

Ignoring the built-in facilities of Jenkins

But we will ignore most of these and leverage the Jenkins pipeline itself through Python code.

Leverage Python as much as possible

Rather than use plug-ins, we will focus on running script code in Python.

The advantage of using Python to do our tasks is that we have full control over the behavior that is needed, minimizing dependencies on other software components inside of Jenkins that were not well docucumented or unclear how to install and use.

It gives us the ultimate flexibility in defining the workflow that we want. The downside of writing Python code to automate the pipeline is that you must know a little bit about writing software. But you could learn everything you need in just a few days.

The Jenkins plug-in model took too much of my time

Frankly, I had a lot of challenges setting up the various plug-in components to work properly with Jenkins in my Linux environment. There was a lot lacking in the documents.

Productive immediately with Python

The moment I switch to a Python based approach, I started to become productive almost immediately. I knew the tooling from the command line point of view so translating commandline using Python is really easy using the subprocess module.

Jenkins can run almost anywhere

Because Jenkins relies on Java, it can practically run on any flavor of Linux. In my case, I will be using Ubuntu, tested on version 14.04 and above.

You need to have a JDK and JRE installed. openjdk-7-jre and openjdk-7-jdk are suggested.

Jenkins – groupadd

I discovered some issues if I didn't add the Jenkins user to the appropriate group. You may or not face this issue.

First create the JENKINS_USER. Then,

groupadd jenkins
useradd -g jenkins jenkins

Docker

Because we will be building docker images, running them as containers, and performing tests against them, we will obviously need the docker client tooling installed on our Jenkins host, which is our Linux VM.

MySQL - How are you going to capture and retain state for your pipeline?

**I debated this quite a bit previously to myself. ** The question becomes, quote how do you track state for the different steps in the pipeline. Obviously if you have the failure in any of the steps, the system needs to react appropriately.

My simple rule was that if there was anything that went wrong, we would write a record into MySQL. That records would then would be read in subsequent steps in the pipeline.

The reason I like this approach is that the data can be somewhat structured and I could easily accessed from anywhere. Debugging the pipeline is a simple SQL statement. The database could point you to the Python script. Because all the details are contained within each Python script, debugging the pipeline could be a breeze.

More options are available for debugging as well. You could have generous print statements, in our Python scripts that then get trapped to Jenkins log files.

Bottom Line - The last thing you want to be debugging is pipeline code.

The point is to resolve as quickly as possible where the pipeline breaks down.

Constantly tracking state in a persistent relational data store is my answer to stay tracking in this situation. you could go really crazy now partying on this relational data. You can use some analytics even machine learning into the whole equation here. You can even use analytics dashboards like Power BI and be able to view all your bills holistically is one big system. Frankly, there will not be much more for you to add after I finish here demonstrating the entire pipeline in all its gory details.

You could also take prolific log files that get generated within Jenkins and persists those to blob storage.

There is a huge amount of intelligence built into the logging mechanism if you do it correctly.

Click image for full size

Figure 4: Tracking pipeline state in a relational database

Installing MySQL

Click image for full size

Figure 5: Installing MySQL

Installing the JDBC Drivers

Click image for full size

Figure 6: x

Hello World

So the next task is to bring Jenkins up and create a project to begin implementation of our pipeline.

There are two aspects to getting Jenkins started.

The first one is to actually run the war file that gets downloaded.

Here you can see the war file is called Jenkins.war.

The command below also assumes that Java is properly installed and available in the path.

Click image for full size

Figure 7: Starting Jenkins on the command line

Starting your browser

The public IP address that you see here in the address bar of the browser is the public IP address of the virtual machine running in Azure.

You can get this public IP address from the Azure portal.

If your VM has a domain name, you could also use that. Notice that I mapped to port 8086.

That's because I passed in the parameter as you see below to indicate the port that I wanted exposed by Jenkins.

java -jar jenkins.war --httpPort=8086

Click "New Item."

Click image for full size

Figure 8: New Jenkins Project

Creating our first pipeline project

After clicking a "new item," the next step is to provide a "name" and select the "pipeline" project type.

Click image for full size

Figure 9: Building our pipeline from the template of Jenkins

The most simple pipeline we could start with is a “hello world” pipeline as seen below.

Click image for full size

Figure 10: Adding Pipeline code

Click image for full size

Figure 11: Build Now

Click image for full size

Figure 12: Viewing the Console Output

Conclusion

The takeaways from this post is that we are up and running with Jenkins on our Ubuntu VM in Azure. A variety of software technologies were leveraged here. And the point was to get them all installed. We also went in and ran a basic hello world pipeline. You also probably notice that we have defined the entire pipeline in terms of Python script. The names chosen were very descriptive so it’s very clear exactly where to go to debugging our pipeline. Over the next few posts on me to reveal the source code that makes this pipeline possible. If you’re interested in that code, just let me know.