Users who have contributed to this file

Kubernetes Local Cluster Experience

This proposal attempts to improve the existing local cluster experience for kubernetes.
The current local cluster experience is sub-par and often not functional.
There are several options to setup a local cluster (docker, vagrant, linux processes, etc) and we do not test any of them continuously.
Here are some highlighted issues:

Docker based solution breaks with docker upgrades, does not support DNS, and many kubelet features are not functional yet inside a container.

Vagrant based solution are too heavy and have mostly failed on OS X.

Local linux cluster is poorly documented and is undiscoverable.
From an end user perspective, they want to run a kubernetes cluster. They care less about how a cluster is setup locally and more about what they can do with a functional cluster.

Primary Goals

From a high level the goal is to make it easy for a new user to run a Kubernetes cluster and play with curated examples that require least amount of knowledge about Kubernetes.
These examples will only use kubectl and only a subset of Kubernetes features that are available will be exposed.

Works across multiple OSes - OS X, Linux and Windows primarily.

Single command setup and teardown UX.

Unified UX across OSes

Minimal dependencies on third party software.

Minimal resource overhead.

Eliminate any other alternatives to local cluster deployment.

Secondary Goals

Enable developers to use the local cluster for kubernetes development.

Non Goals

Simplifying kubernetes production deployment experience. Kube-deploy is attempting to tackle this problem.

Supporting all possible deployment configurations of Kubernetes like various types of storage, networking, etc.

Existing solutions

Spread's UX is great!
It is adapted from monokube and includes DNS as well.
It satisfies almost all the requirements, excepting that of requiring docker to be pre-installed.
It has a loose dependency on docker.
New releases of docker might break this setup.

Single binary that runs all kube master components.
Does not include DNS.
This is only a part of the overall local cluster solution.

Vagrant

The kube-up.sh script included in Kubernetes release supports a few Vagrant based local cluster deployments.
kube-up.sh is not user friendly.
It typically takes a long time for the cluster to be set up using vagrant and often times is unsuccessful on OS X.
The Core OS single machine guide uses Vagrant as well and it just works.
Since we are targeting a single command install/teardown experience, vagrant needs to be an implementation detail and not be exposed to our users.

Proposed Solution

To avoid exposing users to third party software and external dependencies, we will build a toolbox that will be shipped with all the dependencies including all kubernetes components, hypervisor, base image, kubectl, etc.
Note: Docker provides a similar toolbox.
This "Localkube" tool will be referred to as "Minikube" in this proposal to avoid ambiguity against Spread's existing "localkube".
The final name of this tool is TBD. Suggestions are welcome!

Minikube will provide a unified CLI to interact with the local cluster.
The CLI will support only a few operations:

Upgrade - upgrades internal components to the latest available version (upgrades are not guaranteed to preserve cluster state)

For running and managing the kubernetes components themselves, we can re-use Spread's localkube.
Localkube is a self-contained go binary that includes all the master components including DNS and runs them using multiple go threads.
Each Kubernetes release will include a localkube binary that has been tested exhaustively.

To support Windows and OS X, minikube will use libmachine internally to create and destroy virtual machines.
Minikube will be shipped with an hypervisor (virtualbox) in the case of OS X.
Minikube will include a base image that will be well tested.

In the case of Linux, since the cluster can be run locally, we ideally want to avoid setting up a VM.
Since docker is the only fully supported runtime as of Kubernetes v1.2, we can initially use docker to run and manage localkube.
There is risk of being incompatible with the existing version of docker.
By using a VM, we can avoid such incompatibility issues though.
Feedback from the community will be helpful here.

If the goal is to run outside of a VM, we can have minikube prompt the user if docker is unavailable or version is incompatible.
Alternatives to docker for running the localkube core includes using rkt, setting up systemd services, or a System V Init script depending on the distro.

To summarize the pipeline is as follows:

OS X / Windows

minikube -> libmachine -> virtualbox/hyper V -> linux VM -> localkube

Linux

minikube -> docker -> localkube

Alternatives considered

Bring your own docker

Pros

Kubernetes users will probably already have it

No extra work for us

Only one VM/daemon, we can just reuse the existing one

Cons

Not designed to be wrapped, may be unstable

Might make configuring networking difficult on OS X and Windows

Versioning and updates will be challenging. We can mitigate some of this with testing at HEAD, but we'll - inevitably hit situations where it's infeasible to work with multiple versions of docker.

There are lots of different ways to install docker, networking might be challenging if we try to support many paths.

Vagrant

Pros

We control the entire experience

Networking might be easier to build

Docker can't break us since we'll include a pinned version of Docker

Easier to support rkt or hyper in the future

Would let us run some things outside of containers (kubelet, maybe ingress/load balancers)

Cons

More work

Extra resources (if the user is also running docker-machine)

Confusing if there are two docker daemons (images built in one can't be run in another)

Always needs a VM, even on Linux

Requires installing and possibly understanding Vagrant.

Releases & Distribution

Minikube will be released independent of Kubernetes core in order to facilitate fixing of issues that are outside of Kubernetes core.

The latest version of Minikube is guaranteed to support the latest release of Kubernetes, including documentation.

The Google Cloud SDK will package minikube and provide utilities for configuring kubectl to use it, but will not in any other way wrap minikube.