Tooling

News

Resource Lists

Guide

Essentials

Advanced Configuration

Internals

Documentation

While it is extremely easy to set up Keel and just use provided default configuration, sometimes you need additional configuration to better solve your use-case.

Policies

Use policies to define when you want your application to be updated. Providers can have different mechanisms of getting configuration for your application, but policies are consistent across all of them. Following semverbest practices allows you to safely automate application updates.

Available policies:

all: update whenever there is a version bump or a new prerelease created (ie: 1.0.0 -> 1.0.1-rc1)

major: update major & minor & patch versions

minor: update only minor & patch versions (ignores major)

patch: update only patch versions (ignores minor and major versions)

force: force update even if tag is not semver, ie: latest, optional label: keel.sh/match-tag=true which will enforce that only the same tag will trigger force update.

Pre-release tags

According to semver (http://semver.org/) spec, version tags can have additional metadata such as 1.0.0-dev, 1.0.0-prod, etc. Keel deals with semver tags by only allowing updates with the same version metadata.

Example:

tag: 0.5.0-dev policy: minor will only be updated by 0.6.0-dev and not 0.5.0-prod

tag 0.5.0 policy: minor will not be updated by 0.6.0-rc1

This way you can easily separate different environments and update them independently.

Additional settings

Keel tries to mostly rely on your resource configuration files, such as deployment, daemonset, statefulset labels & annotations. Here is an example with all available options:

Providers

Providers are direct integrations into schedulers or other tools (ie: Helm). Providers are handling events created by triggers. Each provider can handle events in different ways, for example Kubernetes provider identifies impacted deployments and starts rolling update while Helm provider communicates with Tiller, identifies releases by Chart and then starts update.

Available providers:

Kubernetes (supports Deployments, DaemonSets, StatefulSets)

Helm

While the goal is often the same, different providers can choose different update strategies.

Kubernetes

Kubernetes provider was the first, simplest provider added to Keel. Policies and trigger configuration for each application deployment is done through labels.

Policies are specified through special label:

keel.sh/policy=all

A policy to update only minor releases:

keel.sh/policy=minor

Kubernetes example

Here is an example application deployment.yaml where we instruct Keel to update container image whenever there is a new version:

Deployment polling example

While the deployment above works perfect for both webhook and Google Cloud Pubsub triggers sometimes you can’t control these events and the only available solution is to check registry yourself. This is where polling trigger comes to the rescue.

Note: when image with non-semver style tag is supplied (ie: latest) Keel will monitor SHA digest. If tag is semver - it will track and notify providers when new versions are available.

Add labels:

keel.sh/policy=force # add this to enable updates of non-semver tagskeel.sh/trigger=poll

Helm

Helm helps you manage Kubernetes applications — Helm Charts helps you define, install, and upgrade even the most complex Kubernetes application. More information can be found on project’s website https://helm.sh/.

Keel works directly with Tiller (a daemon that is used by Helm CLI) to manage release upgrades when new images are available.

Helm example

Keel is configured through your chart’s values.yaml file.

Here is an example application values.yaml file where we instruct Keel to track and update specific values whenever there is a new version:

Triggers

Triggers are entry points into the Keel. Their task is to collect information regarding updated images and send events to providers.

Available triggers:

Webhooks

Native Webhooks

DockerHub Webhooks

Quay Webhooks

Harbor webhooks

Gitlab webhooks

Receiving webhooks without public endpoint

Google Cloud GCR registry

Polling

Webhooks

Webhooks are “user-defined HTTP callbacks”. They are usually triggered by some event, such as pushing image to a registry. Native webhooks (simplified version) are accepted at /v1/webhooks/native endpoint with a payload that has name and tag fields:

{"name": "gcr.io/v2-namespace/hello-world", "tag": "1.1.1"}

Keel by default runs HTTP server on port 9300. Create a service and either expose it to the internet or use https://webhookrelay.com to receive webhooks.

DockerHub Webhooks

DockerHub uses webhooks to inform 3rd party systems about repository related events such as pushed image.

https://docs.docker.com/docker-hub/webhooks - go to your repository onhttps://hub.docker.com/r/your-namespace/your-repository/~/settings/webhooks/ and point webhooksto /v1/webhooks/dockerhub endpoint. Any number of repositoriescan send events to this endpoint.

Quay Webhooks

Documentation on how to setup Quay webhooks for Repository Push events is available here: https://docs.quay.io/guides/notifications.html. These webhooks should be delivered to /v1/webhooks/quay endpoint. Any number of repositoriescan send events to this endpoint.

Google Cloud GCR registry

If you are using Google Container Engine with Container Registry - search no more, pubsub trigger is for you.

You will need to create a Google Service Account to use PubSub functionality.

Since Keel requires access for the pubsub in GCE Kubernetes to work - your cluster node pools need to have permissions. If you are creating a new cluster - just enable pubsub from the start. If you have an existing cluster - currently the only way is to create a new node-pool through the gcloud CLI (more info in the docs):

Update Keel’s environment variables

Make sure that in the deployment.yaml you have set environment variables PUBSUB=1, PROJECT_ID=your-project-id and GOOGLE_APPLICATION_CREDENTIALS to your secrets yaml.

Polling

Since only the owners of docker registries can control webhooks - it’s often convenient to usepolling. Be aware that registries can be rate limited so it’s a good practice to set up reasonable polling intervals.While configuration for each provider can be slightly different - each provider has to accept several polling parameters:

Explicitly enable polling trigger

Supply polling schedule (defaults to 1 minute intervals)

Cron expression format

Custom polling schedule can be specified as cron format or through predefined schedules (recommended solution).

Available schedules:

Entry

Description

Equivalent To

@yearly (or @annually)

Run once a year, midnight, Jan. 1st

0 0 0 1 1 *

@monthly

Run once a month, midnight, first of month

0 0 0 1 * *

@weekly

Run once a week, midnight on Sunday

0 0 0 * * 0

@daily (or @midnight)

Run once a day, midnight

0 0 0 * * *

@hourly

Run once an hour, beginning of hour

0 0 * * * *

Polling with AWS ECR

If you are using polling trigger with Amazon ECR registry, Keel deployment requires several environment variables:

Tip: If you want to disable polling support for your Keel installation - set environment variablePOLL=0.

Approvals

Users can specify on deployments and Helm charts how many approvals do they have to collect before a resource gets updated. Main features:

non-blocking - multiple deployments/helm releases can be queued for approvals, the ones without specified approvals will be auto updated.

extensible - current implementation focuses on Slack but additional approval collection mechanisms are trivial to implement.

out of the box Slack integration - the only needed Keel configuration is Slack auth token, Keel will start requesting approvals and users will be able to approve.

stateful - uses github.com/rusenask/k8s-kv for persistence so even after updating itself (restarting) it will retain existing info.

self cleaning - expired approvals will be removed after deadline is exceeded.

Enabling approvals

Approvals are enabled by default but currently there is only one way to approve/reject updates:Slack - commands like:

keel get approvals - get all pending/approved/rejected approvals

keel approve <identifier> - approve specified request.

keel reject <identifier> - reject specified request.

Make sure you have set export SLACK_TOKEN=<your slack token here> environment variable for Keel deployment.

If you wish to specify a special channel for approval requests, supply SLACK_APPROVALS_CHANNEL=<approvals channel name> environment variable and then invite Keel bot to that channel.

Configuring via Kubernetes deployments

The only required configuration for Kubernetes deployment to enable approvals is to add keel.sh/approvals: "1" with a number (string! as the underlying type is map[string]string) of required approvals.

Slack notifications

First, get a Slack token, info about that can be found in the docs. Then, provide token via SLACK_TOKEN environment variable. You should also provide SLACK_CHANNELS environment variable with a comma separated list of channels where these notifications should be delivered to.

Keel will be sending messages when deployment updates succeed or fail.

Hipchat notifications

Coming soon…

Mattermost notifications

Mattermost is an open source Slack alternative, it’s written in Go and React.

If you don’t have a Mattermost server, you can set one up by using Docker: