Snappy Jenkins CI

This repo helps setting up the CI and CD environment used for Snappy related projects, including https://github.com/snapcore/snapd and https://github.com/snapcore/snapcraft. It contains Dockerfiles which define the required containers (Jenkins master and slaves and a Nginx proxy) and several shell scripts for bringing them to live both in a cloud provider (only OpenStack supported at the moment) and locally.

The testing jenkins jobs use the snappy-cloud-client binary from https://launchpad.net/snappy-tests-job for launching the tests in cloud instances, it takes care of determining the most recent Snappy cloud image for a given channel and release (see below for the required name patterns), instantiating a cloud instance of that image and getting its ip. Then they execute the integration suite in the instance, shut it down when finished or in case of errors and generates reports about the tests executed, including a subunit version of the result sent to a practitest backend and calculations about integration tests coverage.

There are also jobs configured for accessing the Snappy Product Integration environment. This gives a lot more flexibility when executing the tests not being only limited to cloud instances but also on hardware boards, however you must provide credentials for this service.

Once the scripts finish you can check the machine ip executing docker-machine ip snappy-jenkins-local and access the jenkins master instance from the browser at http://<server_ip>:8080

If you want to try the jobs you can download the credentials from an accessible running server, see Initializing servers below.

Cloud provision

In this case you need to have OpenStack credentials loaded, for example by sourcing a novarc file:

$ source path/to/openstack/credentials/novarc

Before executing the provision script you should have at least the $OS_USERNAME and $OS_REGION_NAME environment variables set. This credentials are used for creating the hosts where the CI environment itself is going to run, and may be different to the ones used later for the Ubuntu Core instances, more on this later. The scripts also assume that you have a valid private key at ~/.canonistack/${OS_USERNAME}_${OS_REGION_NAME}.key, if that's not the case you can pass the path as the last argument.

The cloud provision process begins with the creation of the swarm cluster and later provision of the images, for it to work wee need firat to have set up the secrets and key-value backend, which can be done with this command (more on this in the Secrets section below):

$ ./bin/secrets/cloud-provision.sh

Then the swarm cluster can be created and the containers initially deployed with this is done with this single command:

$ ./bin/jenkins/create-cluster.sh

It creates the swarm master, the nodes, pulls the container images, runs compose up and scales the required slaves.

You can retrieve the job history from an accessible running server, and also initialize the required credentials in order to run jobs, see Syncing servers below.

Redeploy

Once the cluster is deployed in order to update the containers you can use the redeploy script:

$ ./bin/jenkins/cloud-redeploy.sh

It backups the jobs' history, stops the running containers, pulls the new images, restarts the cluster, restores the history, injects the credentials and cleans up leftover containers and volumes.

GitHub integration

In cloud deployments, the provision command sets up a security group that allows access to port 8081 from everywhere and ports 8080 and 22 from a local range (by default 10.0.0.0/8). All this configuration is done this way in order to facilitate a secure connection from the GitHub webhook.

First of all, you should assign a floating IP to your VM instance so that it can be reached from GitHub. The container infrastructure and security group assigned make it secure to expose it to the wild, as explained earlier.

The github-snappy-integration-tests-cloud job is configured to receive payloads from GitHub in response to events. It points to the snapcore/snapd repository, you can change this to access one of your own repos to try the hook.

In the settings page of the repository configured in github-snappy-integration-tests-cloud you should setup the webhook to notify Jenkins of changes in the repository, the URL should be http://<floating_ip>:8081/ghprbhook/ and the events to trigger the webhook "Pull Request" and "Issue Comment"

In order to be able to respond to comments in the pull request and update the status of the tests once they are triggered you should setup a bot user to be managed by Jenkins (a regular user works too). This user should be added as a collaborator in the repo and its credentials must be added in the jenkins general configuration (Manage Jenkins -> Configure System -> GitHub Pull Request Builder -> GitHub Auth -> Credentials -> Add).

Required images

With cloud provision, in order to spin up the jenkins host an image with a name of the form "wily-daily-amd64" should be accessible for the current user.

With both kinds of provision, for the jobs to be able to run your openstack user should be able to access snappy images with a name of the form ubuntu-core/custom/ubuntu-rolling-snappy-core-amd64-edge* for the rolling release tests and ubuntu-core/custom/ubuntu-1504-snappy-core-amd64-edge* for the 15.04 release tests. This kind of images can be created with the https://github.com/ubuntu-core/snappy-cloud-image. The configuration can be changed with the release and channel switches in the respective jobs.

Data backup and restore

The job history can be saved and restored using the ./bin/jenkins/backup.sh and ./bin/jenkins/restore.sh scripts. Executing:

$ ./bin/jenkins/backup.sh <environment>

creates a backup.tar.gz file in the current directory with all the existing executions of the jobs, being <environment>local or remote (default), and in both cases the client must be pointing to the matching docker engine.

These backup files can be restored to a running server with:

$ ./bin/jenkins/restore.sh /path/to/backup/file <environment>

Secrets management

We use Hashicorp's https://www.vaultproject.io/ to manage the secrets used by the jenkins deployment. There is a script for setting up this service on a cloud environment, you can provision it with:

being <private_key_path> the ssh key that the slaves will use for creating images and accessing instances created from them, and <openstack_credentials_path> the path of the novarc file needed for accessing the cloud resources where the testbeds will be created. These secrets will be created under the vault path secrets/jenkins/tests/ssh-key and secrets/jenkins/tests/openstack-credentials respectively.

After the provisioning, a vault-remote.txt text file is created in the current directory containing the server keys and root token, keep it safe because they are unique for each deployment.

You can also try the deployment locally with:

$ ./bin/secrets/local-provision.sh

bin/secrets/inject.sh gets the credentials from a vault server (see below) and puts them into a local or remote cluster. In order to execute this script you must be able to authenticate against the Vault server. It is executed with:

$ ./bin/secrets/inject.sh <environment>

with <environment> being either local or remote (default).

Finally, you can also backup and restore the credentials with the ./bin/secrets/backup.sh and ./bin/secrets/restore.sh scripts.