As a security company, FireEye relentlessly protects our customers from cyber attacks. To act
quickly on intelligence and expertise learned, the feedback loop from the front lines to features
and capabilities in software must be small. Jenkins helps us achieve this by allowing us to build,
test, and deploy to our hardware and software platforms faster, so we can stop the bad guys
before they reach our customers.

More capabilities and functionalities in our product offerings means more applications and
systems, which means more software builds and jobs in Jenkins. Within the FaaS (FireEye as a
Service) organization, the tens of Jenkins jobs that were manageable manually in the web GUI
quickly grew to hundreds of jobs that required more automation. Along the way, we outgrew
our old legacy datacenter and were tasked with migrating 150+ Freestyle jobs on an old 1.x
Jenkins instance to a newer 2.x instance in the new datacenter in 60 days.

Copying Freestyle job XML configuration files to the new server would leave
technical debt. Using Freestyle job templates would be better but for
complicated jobs that require multiple templates, this would still create large
dependency chains that would be hard to trace in the log output. Finally,
developers were not excited about having to replicate global changes, such as
add an email recipient when a new member joins the team, across tens of jobs
manually or using the
Configuration
Slicer. We needed a way to migrate the jobs in a timely fashion while getting
rid of as much technical debt as possible.

Jenkins Pipeline to the rescue! In 2.0, Jenkins added the capability to create pipelines as first-
class entities. At FireEye, we leveraged many of the features available in pipeline to aid in the
migration process including the ability to:

create Pipeline as Code in a Jenkinsfile stored in SCM

create Jenkins projects automatically when new branches or repos get added with a Jenkinsfile

continue jobs after the Jenkins master or build agent crashes

and most importantly, build a Pipeline
Shared Library that keeps projects
DRY and
allows new applications to be on boarded into Jenkins within seconds

However, Jenkins Pipeline came with a DSL that our users would have to learn to translate their
Freestyle jobs to pipeline jobs. This would be a significant undertaking across multiple teams
just to create Jenkins jobs. Instead, the DevOps team identified similarities across all the
Freestyle jobs that we were migrating, learned the Jenkins DSL to become SMEs for the
organization, and built a shared library of functions and wrappers that saved each Dev/QA
engineer hours of time.

Below is an example function we created to promote builds in Artifactory:

Rather than learning the Jenkins DSL and looking up how the Artifactory Plugin worked in
Pipeline, users could easily call this function and pass it parameters to do the promotion work
for them. In the Shared Library, we can also create build wrappers of opinionated workflows,
that encompasses multiple functions, based on a set of parameters defined in the Jenkinsfile.
In addition to migrating the jobs, we also had to migrate the build agents. No one knew the
exact list of packages, versions, and build tools installed on each build server, so rebuilding
them would be extremely difficult. Rather than copying the VMs or trying to figure out what
packages were on the build agents, we opted to use Docker to build containers with all
dependencies needed for an application.

As we look ahead at FireEye, I will explore how the Shared Library sets us up for easier future
migrations of other tools such as Puppet, JIRA, and Artifactory, and easier integration with new
tools like Openshift. I will also cover our strategies for deployments and plans to move to
Declarative Pipeline.

Alvin will be
presenting
more on this topic at
Jenkins World in August,
register with the code JWFOSS for a 30% discount off your pass.

About the Author

R. Tyler Croy

R. Tyler Croy has been part of the Jenkins project for the past seven years.
While avoiding contributing any Java code, Tyler is involved in many of the
other aspects of the project which keep it running, such as this website,
infrastructure, governance, etc.