Running Spinnaker on Compute Engine

This solution describes best practices for running
Spinnaker on Google Compute
Engine, such as deploying, securing and maintaining your Spinnaker deployment.

When you need to set up a continuous delivery pipeline, Spinnaker provides the
following benefits:

Spinnaker is built on the principle of immutable infrastructure, providing a
foundation for fast, reliable deployments of cloud-native apps.

Deployments are predictable, safe, and can be easily rolled back when needed.

You can create complex deployment pipelines that span multiple development
environments such as testing, QA, and production.

About Spinnaker

Spinnaker is an open source tool for orchestrating continuous delivery pipelines
that deploy software to cloud resources. The codebase is written in Java and
Groovy, and leverages the
Spring Boot framework.
Spinnaker provides an intuitive user interface, and a flexible REST API for
fine-grained control over tasks in each pipeline stage and the dependencies
between stages. With Spinnaker, it's easy to map your software release process
to a set of discrete steps that reliably delivers your software to end users.

The following image describes Spinnaker's release process.

Software is built, then tested. If all tests pass, an immutable image is
"baked" and made available in the cloud. After the image is available it can be
deployed to sets of instances to update their running software.

Spinnaker is an application stack that includes microservice components, each
handling a discrete piece of functionality. Each component generates logs in a
separate directory in the top-level /var/log/spinnaker directory.
Spinnaker's component configuration files
exist in the /opt/spinnaker/config/ directory.

The following components interact directly with Google Cloud Platform (GCP):

The Clouddriver component interacts with the Compute Engine API to deploy
applications and update their load balancers.

The Front50 component handles persistent storage for Spinnaker-specific
configuration and metadata such as pipeline definitions. Configuration data is
stored in Google Cloud Storage.

Deployment architecture

Each major component such as Spinnaker, Jenkins, and Redis runs on its own
virtual machine for independent scaling. Each virtual machine runs in its own
instance group to ensure that at least one instance runs at all times. Using
this setup ensures that if a machine fails or terminates, the Compute Engine
instance group manager brings up another instance in its place.

A distinct deployment network and subnetwork isolates Spinnaker from the rest
of the project’s infrastructure. Static IP addresses keep each component at a
well-known network location so that other components can access them reliably.
The network firewall restricts access except for traffic that Spinnaker needs to
operate, and for administrators to manage Spinnaker. Inside Spinnaker’s network,
only the Spinnaker instance can access Jenkins and Redis.

The architecture uses Cloud Storage to store Spinnaker’s configuration settings,
because Cloud Storage is highly available and provides data replication.

Configuring Spinnaker

Configuration files

Each Spinnaker component has a corresponding configuration file that customizes
its functionality. For example, you can configure the orchestration component
Orca to
run a generic script stage in Jenkins,
which allows you to have a
database migration stage in your pipeline that is defined by a Bash script
within your source code repository. After installing Spinnaker, you can find
these configuration files in the /opt/spinnaker/config directory.

The spinnaker-local.yml file unifies common parameters between components so
that common values can be reused by different components.

The following diagram illustrates the hierarchy of Spinnaker's configuration
files.

You can set environment variables in the /etc/default/spinnaker directory for
certain configuration options in the spinnaker-local.yml file. To see which
options can be set using environment variables, look for variables represented
by the following notation.

${VARIABLE:default-value}

If you don't override these variables, default-value is used.

Create environment variables as key-value pairs separated by an equals sign, as
shown in the following examples.

Configuring image creation

One of Spinnaker’s critical use cases is orchestrating pipelines that deploy
immutable images to VM clusters. Immutable images provide several benefits.

You can roll back the images to specific, previous releases.

Scaling an existing version produces an identical copy of the application.

Your application can boot faster because of minimal startup steps.

Spinnaker leverages Packer to create
immutable images. The Rosco
component manages the image build process. Rosco is configured with a default
Packer configuration template,
located in the /opt/rosco/config/packer/gce.json file. This configuration can
be overridden when creating pipelines by configuring an explicit configuration
file. If you'd like to change the default template, you can either update the
gce.json file, or
create a configuration file
and specify it when configuring the bake stage of your pipeline.

Configuring triggers

Spinnaker uses the Igor
component as its interface with continuous integration (CI) systems such as
Jenkins and Travis CI, and Git repositories such as Github and Stash. The
component handles polling pipeline triggers, and triggering external jobs as
part of a pipeline. Igor can be configured to interface with multiple
independent CI systems for distinct build and integration testing. For pipelines
to be triggered by a repository push, Igor must be configured to poll a CI job
that in turn polls the Git repository, or must be
configured to poll Github or Stash
directly.

Configuring notifications

Spinnaker can send notifications
based on the triggering or results of a pipeline stage. You can configure the
echo component to send
a notification through email or SMS messages, or to Slack or Hipchat.

Security and authorization

This section describes security and authorization best practices for
Spinnaker on GCP.

Authorizing Spinnaker’s access to GCP

You can authorize Spinnaker's access to GCP for deploying to a single or
multiple projects.

Deploying to a single project

When deploying to a single project, Spinnaker can use the default service
account to get credentials for Compute Engine requests. The Spinnaker VM
requires the following IAM roles.

Service Account Actor

Compute Instance Admin

Compute Network Admin

Compute Storage Admin

Storage Admin

Deploying to multiple projects

Many customers choose to run one Spinnaker deployment per GCP project, but if
this use case doesn’t work for you, you can authenticate the Spinnaker service
account for multiple projects. When using
the Deployment Manager templates,
the Spinnaker VM launches with the default service account
that belongs to the project. Complete the following tasks to give Spinnaker
access to deploy to multiple projects.

Authenticating and authorizing users for the UI

You can configure Spinnaker
to use Google OAuth
to authenticate users. When a user first accesses the UI, they are prompted to
authenticate with a Google account, and then forwarded on to the UI. Accounts
with 2-factor authentication enabled are
required to use both forms of authentication to be granted access to the UI.

G Suite customers can help secure their Spinnaker installation by
implementing authorization rules
based on the groups in their domain. Spinnaker allows you to define a list of
groups that a user must belong to for their modifications to be accepted. By
default, all users are allowed to view all resources in the UI. When an account
has group membership requirements, only members of that group are permitted to
make modifications to resources in that account. For example, you can configure
a group of users named "spinnaker-admins" that have the ability to:

Configuring network access

You usually don't need Spinnaker to be accessible from the Internet, because
users typically access Spinnaker through your on-premises network. One
way to access Spinnaker from your on-premises environment is to
set up a Google Cloud VPN instance in the network
where you’ve deployed Spinnaker. The following image describes the setup.

Google Cloud VPN provides an encrypted tunnel for traffic between
your environment and Compute Engine instances, allowing anyone on your corporate
network to access Spinnaker’s private IP address.If you launch Spinnaker without
a public IP address, you need to
create a NAT gateway
that can be used to reach the Internet for repository and dependency downloads.

For situations where only several users interact directly with Spinnaker, you
can set up SSH port forwarding from each user’s workstation to the Spinnaker
instance. Forward the Deck (9000), Gate (8084), and Rosco (8087) ports to use
Spinnaker from a remote machine. For more information about how to set up port
forwarding, see
port forwarding over SSH.

You can lock down internal network traffic by creating firewall rules that
prevent unnecessary access between VMs. Allow traffic only from Spinnaker to
Redis, and from Spinnaker to Jenkins. Limit SSH traffic to the Spinnaker VM.

Upgrading Spinnaker

Upgrade Spinnaker regularly to receive the latest feature enhancements and bug
fixes. To upgrade your Spinnaker installation to the latest release, run a
package upgrade. For example, you can run the following commands to update
Ubuntu 14.04.

Note that restarting Spinnaker terminates any currently running pipelines, and
potentially restarts them when Spinnaker is back up and running.

Backing up and restoring Spinnaker

Spinnaker stores important data such as pipeline, application and project
configurations in Cloud Storage. This data is highly durable, replicated across
regions by default, and encrypted at rest. Spinnaker uses Cloud Storage's
versioning capabilities to enable a pipeline
revision history. The Spinnaker user interface shows the differences between
your pipeline versions, and allows you to restore a previous state.

To backup the data, use the gcloud sync command to replicate the data to a
bucket in another region. To restore the data, sync the data back to a bucket
in a region where you intend to redeploy Spinnaker.

Spinnaker uses Redis as ephemeral storage to keep track of ongoing pipeline
tasks. It isn’t critical to backup this data because pipelines can be restarted
if the state is lost. Losing the Redis state can cause pipelines to
automatically retrigger. Ensure that your deployment pipelines have gating for
important or destructive stages by using manual judgement stages and Spinnaker's
built in notifications. You can have multiple
Redis instances to
provide high availability, in case one host goes down. If you choose to backup
this data, you can configure Redis to periodically take state snapshots and
transfer the files to Cloud Storage. For more information about Redis
persistence and disaster recovery, see
Redis Persistence.