Installing OSS Spinnaker in Kubernetes

Note: If you’re using Armory Spinnaker, a lot of the items in this document are handled automatically for you.

Installing Spinnaker

This document will guide you through a manual installation of Spinnaker (either Open Source or Armory), in your Kubernetes cluster.

Installation Architecture

Here are some things to keep in mind when installing Spinnaker:

Spinnaker is composed of set of microservices (mostly Java-based)

With this installation method, we will choose a namespace within your existing Kubernetes cluster and run the Spinnaker components as Kubernetes objects within your Kubernetes cluster. This actually consists of the following:

Kubernetes secrets, which have Spinnaker configurations and credentials

Kubernetes deployments, which create replicasets which create the pods that actually comprise the Spinnaker components

Kubernetes services, which set up ClusterIP load balancers for the Spinnaker microservices to talk with each other.

Think of the Kubernetes cluster where Spinnaker is installed as the Installation Target

Spinnaker is used to deploy to multiple cloud environments, including Kubernetes. Think of these as Depoyment Targets.

The cluster where Spinnaker is installed (the Installation Target) will also be one of your Deployment Targets.

Spinnaker will need to be able to access the API for each of your Deployment Targets, and will need credentials (either provided directly or via inherited instance permissions) for each of your Deployment Targets (including the Installation Target)

Spinnaker is configured and managed with a tool called Halyard, which consists of the following:

A Halyard daemon, which listens for API calls from the Halyard CLI and actually creates the Kubernetes objects that make up Spinnaker.

A Halyard CLI tool (hal), which is used to interact with the Halyard daemon

You use commands such as hal config provider kubernetes enable to make changes to the Halyard configuration and hal deploy apply to apply and deploy your changes to the KUbernetes cluster.

A yaml-based Halyard configuration file (~/.hal/config, also known as the halconfig), which has the configuration for your cluster

A set of additional yaml files within the ~/.hal directory and subdirectories, which can be used to customize your Spinnaker installation

In addition to the Kubernetes cluster where Spinnaker is installed, Spinnaker needs the following persistent state stores:

An S3 or S3-compatible endpoint (S3, GCS, AZS, Minio, etc.) in which to store persistent configuration.

This can be internal or external to the Kubernetes cluster; in this document, we will use an S3 bucket.

A Redis or Redis-compatible endpoint (Elasticache, Memorystore, etc.) both to use for execution history and to use as a cache.

This can be internal or external to the Kubernetes cluster; in this document, Halyard will default to creating a redis container alongside the Spinnaker microservices, and Spinnaker will use that.

Halyard can be run in multiple places; it only needs the following:

API access to your Kubernetes cluster

Credentials for your various cloud environments.

In this document, we will run Halyard in a Docker container from outside the cluster, and it will interact with your pre-existing Kubernetes cluster.

Halyard can be run locally on your workstation, or on some persistent virtual machine such as an EC2 instance that has access to your Kubernetes cluster.

You should both preserve and keep secret the full contents of everything in your .hal directory and all other directories mounted into your Halyard container.

This document will guide you through the initial installation of Open Source or Armory Spinnaker, in the following environment:

Spinnaker installed in Kubernetes

Using an S3 bucket as the configuration store

Installed via Halyard, run as a Docker container

This halyard container will have multiple volume mounts added to it, discussed below.

We’ll do this by starting a Halyard docker container (with a container name of oss-halyard), and then interacting with the Halyard daemon in that container from another shell session.

Later articles may cover the following:

Configuring Spinnaker to use an external Redis (rather than the default redis Kubernetes container)

Pre-requisites

You must have the following items available prior to installing Spinnaker:

A place to run Halyard. This can be any place a Docker container can run. Here are some options:

On a Linux or OSX workstation (Windows should work, but permissions for volumes can be complicated)

On an EC2 instance with Docker installed.

An S3 bucket, and IAM permissions to list the bucket and read/write to the S3 bucket, via one of the following:

IAM credentials in the form of an AWS Access Key and Secret Accesss Key

IAM instance role attached to the Kubernetes nodes where Spinnaker will be running

Credentials to your Kubernetes cluster (a kubeconfig) that can be used from within a Docker container. If you’re using IAM credentials for an EKS cluster, we will copy these IAM credentials into the Docker container (in ~/.aws), so EKS credentials should work.

Create the bucket

Log into your AWS account, and navigate to S3

Click “Create Bucket”

In “Bucket Name”, specify the name of the S3 bucket you’d like to use. Click “Next”

Under “Configure Options”, select “Keep all versions of an object in the same bucket.” and “Automatically encrypt objects when they are stored in S3.” (and any other options you would like to use). Click “Next”

Under permissions, select options to keep the bucket as restricted as possible. Click “Next”

Click “Create Bucket”

Create an IAM policy for the bucket

In your AWS account, navigate to IAM

Click “Policies” (on the left nav bar)

Click “Create policy”

Click “JSON”.

In the JSON field editor, add this (replace BUCKET_NAME with your bucket name):

Installation

The next several sections will walk through the installation of Spinnaker on your Kubernetes cluster.

Start the Halyard container

Halyard can be run directly on a Linux machine or on OSX, but is somewhat more portable in a Docker container. In order to to this, you should directly mount three directories into your docker container. It’s useful to have these directories accessible from a shared root. We also recommend adding an additional directory to store sensitive information passed to Spinnaker (Spinnaker secret management forthcoming):

Then, in a separate terminal/shell session, enter the docker container with this:

docker exec-it halyard bash
# Also, once in the container, you can run these commands for a friendlier environment to:# - prompt with information# - alias for ls# - cd to the home directoryexport PS1="\h:\w\u\$ "alias ll='ls -alh'cd ~

Create a service account with which to install Spinnaker

Halyard will install Spinnaker in your Kubernetes cluster; it is a good idea to have a dedicated service account for Halyard to use to interact with your Kubernetes cluster. First, we’ll create the namespace where Spinnaker will live, and then create a service account and permissions to install Spinnaker into that namespace.

Set up bash parameters in the container

First, we’ll set up bash environment variables that will be used by later commands

Do this in the Halyard container

# Specify the name of the kubernetes context that has permissions to create the service account in your# target cluster and namespace. To get the list of contexts, you can run "kubectl config get-contexts"export CONTEXT="aws-armory-dev"# Enter the namespace that you want to install Spinnaker in. This can already exist, or can be created.export NAMESPACE="spinnaker-installation"# Enter the name of the service account you want to create. This will be created in the target namespaceexport SERVICE_ACCOUNT_NAME="spinnaker"# Enter the name of the role you want to create. This will be created in the target namespaceexport ROLE_NAME="spinnaker-role"# Enter the name you want Spinnaker to use to identify the cloud provider accountexport ACCOUNT_NAME="spinnaker"# Replace with the bucket you are usingexport BUCKET_NAME=justinrlee-spinnaker-temp
# Replace with the AWS region where your bucket isexport REGION=us-east-1
# This is the default directory within the bucket that will be used by Spinnaker; feel free to change to a different directory.export ROOT_FOLDER=front50

Create the namespace to install Spinnaker in

If the namespace does not exist, you can create it.

Do this in the Halyard container

kubectl --context${CONTEXT} create ns ${NAMESPACE}

Create the service account and credentials

Create a Kubernetes manifest that contains the service account, role, and rolebinding.
This will create two roles within the specified namespace, one with full access to the namespace and one with limited access to the namespace. It will also bind both roles to the service account. In the next step, we show how to remove the admin role binding, if you want fewer permissions

Optionally, remove the admin role binding

The above set of roles and bindings will grant your service account full access to all items with the Spinnaker namespace. If you don’t want this set of permissions, you can remove the admin role binding:

Select the version of Spinnaker to install

Before Halyard will install Spinnaker, you should specify the version of Spinnaker you want to use.

You can get a list of available versions of spinnaker with this command:

hal version list

And then you can select the version with this:

# Replace with desired version from output above:# Armory versions will start with 2.x.x# OSS will start with 1.x.xexport VERSION=2.1.3
hal config version edit --version$VERSION

Install Spinnaker

Now that your halconfig is completely configured for the initial Spinnaker, you can tell Halyard to actually install Spinnaker:

hal deploy apply

Connect

It may take a little bit of time for your Spinnaker installation to complete. You can monitor by running:

kubectl get pods -n${NAMESPACE}--watch

In order to connect to your Spinnaker cluster, you can port forward directly from your workstation to your spinnaker cluster. This must be run on your local workstation. You must forward port 8084 to spin-gate (Spinnaker’s API) and port 9000 to spin-deck (Spinnaker’s UI)

Set up DNS so that your two URLs point to the IP address for the ingress (in the above, configure spinnaker.some-url.com and api.some-url.com to point to 35.233.216.189). This can be done via whatever your organization uses for DNS.

Configuring TLS Certificates

Configuration of TLS certificates for ingresses is often very organization-specific. In general, you would want to do the following:

Add certificate(s) so that your ingress controller can use them

Configure the ingress(es) so that NGINX (or your ingress) terminates TLS using the certificate(s)

Update Spinnaker to be aware of the new TLS endpoints (note https instead of http)