04.08.2017Building foreman RPM packages using docker container

The ATIX development team spent a lot of time in the last Scrum improving the build environment of orcharhino. So far we've been using Jenkins build jobs, which did the job already pretty well. However this approach is not always as flexible as we need it for maintaining a valuable product delivery for our customers. The Jenkins build steps were mainly shell scripts which were directly configured in the Jenkins UI.

The build process itself utilised Docker-containers. Since the whole structure grew historically, we had a lot of duplicates for nearly identical tasks, generating nearly identical Dockerfiles. This was not a big issue in general, unless we had to make amendements. Depending on the necessary changes this could affect a bunch of files at once, causing a whole series of necessary changes and waste of time and effort. We needed to change and improve this!

After playing around with various sandbox environments based on Jenkins pipelines, we came to the conclusion that moving towards the Jenkins pipeline way, i.e. a "code-based" build environment would be the most elegant solution. As we knew that the upstream Foreman team is successfully using tito / mock to build Foreman, we first took this tool into consideration. However, our conclusion was that building the RPM packages within Docker is at least as good as the tito / mock tool chain.

On top it has the advantage of creating the building environment only once from scratch and reusing it, rather than recreating it for every package which is in the build process. Overall this means we can decrease the RPM building time, using a Docker based approach.

The challenging parts of creating Foreman packages are as follows:

there are many different Foreman packages

multiple git repositories: The RPM spec file and some resources are stored in foreman-packaging, however the main source code is in the foreman git repository

the Foreman build requires a lot of dependencies

The objectives we wanted to achieve with a Jenkins pipeline approach, utilising Docker were:

Speed up the overall building process for one specific orcharhino release

Everything as code - even the build instructions

Reuse the good code from the old Jenkins build steps and rewrite some bad scripts

That's all. After the build is done, the RPMs are located in the directory "foreman/RPM". How does it work internally?

After cloning the foreman-packaging and the Foreman repository the script collects all necessary build files and moves them to foreman/_pkg_build.

Then the interesting part happens.

The script uses erb (embedded ruby) to dynamically create a Dockerfile and the build_rpm.sh script. Those two files are stored in foreman/_pkg_build. The build_rpm.sh is later used inside the Docker container to build the RPM. The Dockerfile contains the build instructions for the Docker container. Since the script is able to create other Foreman packages from the Foreman-packaging distribution as well, the Dockerfile is dynamically generated, based on variables which are located in foreman/_pkg_build/docker_vars.rb. This latter file is written by the build_foreman_packages.sh script.

The next step is to build the Docker container using the Dockerfile and some files in foreman/_pkg_build.Finally, the Docker container is started to build the RPM. This might take some time, since now the build dependencies from the foreman.spec file are evaluated and installed within the Docker container. In the last step,the RPM packages are build and stored in "foreman/RPM".