How we migrate 350+ Maven CI jobs to Pipeline as Code with Jenkins 2 and Docker!

A year ago at eXo, we decided to build all our projects in Docker containers.

In this series of articles, we will tell you our story of how we upgraded more than 350 Jenkins “standard” Maven jobs to Pipeline as code on our Continuous Integration servers, using Jenkins 2 and Docker.

This is a good opportunity to revisit the problems we encountered and the solutions we adopted, as well as to examine some best practices around Maven/Gradle/Android builds in Docker containers, all managed by the Jenkins Pipeline as Code pattern.

As with all important technical migration, we did it step by step. The 3 major steps were:

Create CI Docker Images

Use Jenkins Pipeline & Pipeline Docker plugins

Generate all Pipeline jobs with DSL jobs

The following diagram represents the workflow for these steps:

In this article, we will explain the context (why we did it), and the first step of the migration towards creating your own CI Docker images.

What we build

eXo Platform is built upon open-source and open standards. It adheres to the the Java EE stack and leverages many open-source components.

We have to manage many builds on our CI servers for a variety of reasons, which we can divide into several themes:

Platform components and add-ons

As the first step we have multiple git repositories for all the components required to create a distribution package for eXo Platform:

25+ components to build

Platform projects (cf. diagram below)

Juzu projects

Add-ons

15+ supported Add-ons

100+ community Add-ons

Native Mobile projects

Android Application

iOS Application

Git Workflow

We used a Git Workflow, based on a branching model. All eXo projects are required to follow this model;

Platform versions and clients

Over the last 10 years eXo has released many versions of the eXo Platform. With eXo Platform 5 currently in development, in addition to the versions we maintain for our clients (eXo Platform 4.x : 4.0, 4.1, 4.2, 4.3, 4.4), we always need to manage several versions of the Build stack.

We build projects with the JDK6, JDK7 or JDK8, and Maven 3.0, Maven 3.2 or Maven 3.3 versions. The native Android application is built by Gradle.

Remember how to create Jenkins job via the UI?

Even if Jenkins 2 has benefited from the many efforts to enhance it, it’s always painful to create a new Maven job via the UI, as you can see below:

“Jenkins DSL and Pipeline jobs with Docker to the rescue!”

As a result, we have had to manage a lot of builds via the Jenkins UI and many tools in several versions (Maven, JDK, etc.) on CI agents and development laptops.

Finding this inefficient and painful, we decided to find a solution to automate and manage it in a different way.

You willl notice that we define the EXO_CI_USER_UID variable with the Docker ARG instruction. This is important for the developer experience, which we will explain later in this series of articles.

ENTRYPOINT and CMD

In a Dockerfile, the ENTRYPOINT instruction is an optional definition for the first part of the command to be run. So both ENTRYPOINT or CMD instructions, specified in your Dockerfile, identify the default executable for the Docker image.

The best way is to combine both of them by using CMD to provide default arguments for the ENTRYPOINT.

In the earlier version of the Jenkins Docker Pipeline plugin, Jenkins didn’t use the –entrypoint option when starting containers, so the command was:

1

$docker run...-e ********exoplatform/ci:jdk8-maven33 cat

In that case, if your Docker Image declared an ENTRYPOINT as startup command, Jenkins was not able to use your image.

That’s why we added a custom script in our Docker Images, as a workaround, which allows one to execute Maven as a startup command, but also to execute the cat command and other commands executed with an absolute path:

Dockerfile

1

2

3

4

5

COPY docker-entrypoint.sh/usr/bin/docker-entrypoint

# Workaround to be able to execute others command than "mvn" as entrypoint

ENTRYPOINT["/usr/bin/docker-entrypoint"]

CMD["mvn","--help"]

docker-entrypoint.sh

1

2

3

4

5

6

# Hack for Jenkins Pipeline: authorize cat without absolute path

if[["$1"=="/"*]]||[["$1"=="cat"]];then

exec"$@"

fi

exec mvn"$@"

In the latest versions of the Jenkins Pipeline Docker plugin, this problem has been fixed, to allow use of the –entrypoint option so that the ENTRYPOINT instruction is always overridden:

1

$docker run...-e ********--entrypoint cat exoplatform/ci:jdk8-maven33

Use inheritance to avoid code duplication

We have created and continue to create Docker images to cover our entire Build Stack. Currently this list contains:

exoplatform/ci:jdk6-maven30

exoplatform/ci:jdk7-maven30

exoplatform/ci:jdk7-maven32

exoplatform/ci:jdk8-maven32

exoplatform/ci:jdk8-maven33

exoplatform/ci:jdk8-gradle2

…

As you can imagine, there are not many differences between all those images, so we have created CI base images at several levels to be able to avoid code duplication as much as possible.

The diagram below shows how those images are organized:

Below is an extract of the Dockerfile for the exoplatform/ci:jdk8-maven33 CI Image. You can see that there are only Docker instructions related to the Maven installation, since all other configurations have been done in the inherited images (define locale, create user CI…).

# Workaround to be able to execute others command than "mvn" as entrypoint

ENTRYPOINT["/usr/bin/docker-entrypoint"]

CMD["mvn","--help"]

As explained before, we combined ENTRYPOINT and CMD Docker instructions to be able to run all Maven commands easily in this container:

1

docker run...-e ********exoplatform/ci:jdk8-maven33 clean package

We also add a custom script, to be able to run the cat command and commands other than Maven by giving the absolute path to the command:

1

docker run exoplatform/ci:jdk8-maven33/bin/echo hello

Finally, you will have noticed that we defined the M2_REPO and MAVEN_OPTS environment variables as Docker ENV instructions with all important parameters having default values, which can be overridden by the -e option in the docker run command.

Test your Docker Images

As with other source code, you can create tests suites for your Docker files and images. For the eXo CI Docker images, we use Goss via dgoss.

Goss is a YAML based tool for validating a server’s configuration

dgoss is a wrapper around Goss that aims to bring the simplicity of Goss to Docker containers.

The first step is to create a YAML file to describe what you want to test in your Docker container. There are some examples online that can help you to create this configuration file, but it can be also be generated from the goss command line.

For example, say we want to check that some Maven configuration files exist in the container. We want to test that the mvn –version command is compliant with the Maven and JDK versions installed.

goss.yaml

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

file:

/home/ciagent/.m2/repository:

title:Validating the presence Maven repository folder

exists:true

/home/ciagent/.m2/settings.xml:

title:Validating the absence of eXo USER settings file

exists:false

/usr/share/maven/conf/settings.xml:

title:Validating the presence of eXo GLOBALsettings file

exists:true

package:

git:

installed:true

title:Check that git isinstalled

command:

mvn--version:

exit-status:0

stdout:

-3.3.9

-1.8.0

-"Default locale: en_US, platform encoding: UTF-8"

stderr:[]

timeout:0

Then to execute those tests, it is a simple matter of executing the dgoss tool:

1

dgoss run-it exoplatform/ci:jdk8-maven33 cat

Conclusion

If you are interested in using or testing these Docker Images, you will find:

Next step

Now that we have created Docker Images for all our Continuous Integration stacks, we will explain in the next article (Part 2) how to use them on your laptop, whatever your operating system. Stay tuned!

I'm the software factory manager at eXo. I'm responsible for all software factory services (Jenkins CI, GitHub, Nexus, Sonar...) and the eXo release process.
I help to make the eXo developer experience better by simplifying and automating as much of the process as possible.
Passionate about IT and open source, I contribute to open source projects in different ways (speaking at conferences, writing blog posts...). Also, I have written a book about Apache Maven 3.