INSTALLING KUBERNETES MANUALLY

2017-09-13

The procedure described here applies to centos but the same recipe can obviously be adapted for other distritions with some
minor tweaks.

I am deploying a 3 nodes cluster and all servers are master/nodes at the same time. This is not recommended by
the k8s team but for a lab environment it is perfectly fine. If you understand this procedure well, then you will find
that deploying nodes and masters separately can be just as easy. Let's assume I have 3 servers with IPs 192.168.1.100,192.168.1.101
and 192.168.1.102.

System preparation

First thing we need to do is disable selinux, disable the firewall, NetworkManager and setup a yum repo.
This is probably a bad idea, but then again, it makes things easier in a lab environment.

If you are behind a corporate proxy, also add this line to that last repo file: proxy=http://yourproxy_address

Install docker

yum install -y docker
systemctl enable docker
systemctl start docker

If you are behind a corporate proxy, add this line to /etc/sysconfig/docker: HTTP_PROXY=http://yourproxy_address
and then restart docker

Install etcd

etcd should be installed on all masters. You actually have to install etcd on the same server where the k8s binaries
will be found, but you should definitely install several instances of it and cluster them. Some people also
suggest to install at least one instance somewhere in the cloud. This is because etcd will store all the persistant
admin stuff that k8s needs. So it would be painful to lose that data. Bringing back (or adding) a k8s node is very easy
and transparent as long as your etcd cluster is intact.

The ETCD_INITIAL_CLUSTER line should list all the hosts where etcd will be running. The other lines where you see 192.168.1.100
should be modified to match the IP address of the server you are currently installing etcd on. and ETCD_NAME should also match
the server you are installing on (see the first line where these names are used). The ETCD_NAME can be any arbitrary name (as long
as you properly match them in ETCD_INITIAL_CLUSTER) but most people try to use the server's hostname.

After having installed etcd on all your servers, wait for the cluster to be up checking the output of "etcdctl cluster-health"
Make sure the cluster is up and healthy before continuing.

Install Flannel

Your k8s cluster will need an overlay network so that all pods appear to be on the same layer 2 segment. This is nice because even though
you cluster runs on different servers, each pod will be able to talk to each other using a virtual network that sits on top of your real
network and will span accross all k8s nodes. It's basically a virtual distributed switch, just like openvswitch does. Flannel is
one such driver that enables that for kubernetes. It is the most basic overly driver and works just fine. For more advanced stuff, Nuage is
an option and there are many other options. If you are new to this (like I was), then this is the SDN stuff that all the cool kids
are talking about.

Kubernetes can be downloaded as a binary package from github. What I really like about these binaries is that they are simple standalone
applications. You don't need to install a RPM and a whole bunch of libraries. Simply copy the executable and run it. You will need a total
of 6 process to run. So the first thing to do is to unpack the binaries. Download the version you want from
https://github.com/kubernetes/kubernetes/releases

Each of the daemons can simply be started from the CLI with a whole bunch of command line arguments, you don't even need any
configuration files. This is beautiful because it is so easy.
So technically, you could just do:

And that's it, kubernetes is running. Simple. But let's get systemd to manage those process. If you are not using systemd (slackware?) then
you can setup a bunch of rc scripts to launch those. But for this example, let's create some systemd unit files that will launch this for us.

Note how I did not hard code all command line arguments in the systemd unit files. Instead I will store them in separate environment files
under /etc/kubernetes. So earlier I was praising about how nice it was to just be able to launch the processes on the command line with not
extra config files or no service files but here I am create service files and config files. I know... I just like the fact that I can
customize it any way I want. So here are the config files needed. But as I said, you could just write one bash script that invokes
all those 5 process with all their command line arguments and you would have zero config file and no need for systemd. just 5 binaries
that you copied on your server.

Then you can enable and start all those new services. You have a k8s cluster running. You can test by invoking "kubectl get nodes"

Some notes about the k8s processes

kubectl can be copied on any machine. It will try to communicate to kubernetes through the apiserver process on the localhost. If
you are running this tool on a server where apiserver is not running, then you need to specify --server=http://api-server-address:8080.
In our case, we have installed the apiserver on all k8s nodes. So you can connect to any instances in the cluster.

The apiserver process needs to run on all masters. This is so you can control each masters remotely. I've configured each
apiserver to only talk to it's local etcd because we have installed etcd on all nodes. But would could configure it to talk to
all etcd servers, even remote ones.

the kube-proxy should run on all worker nodes (in our cae, workers and masters are the same). Let's say you have a pod running a web
server on port 242. You don't know on which node your pod will run. But you want to be able to access it using the IP of any of the nodes
in the cluster. That is what kube-proxy does. So in you go to http://192.168.1.100:242 but your pod runs on 192.168.1.102, then kube-proxy
will handle it. It will, as you guessed it, proxy the request. And this is at the TCP/UDP level, it is not just a http proxy.

I am not entirely sure which of these processes are considered as "master" vs "worker nodes" processes. But I believe
that the nodes should only be running kube-proxy and kubelet. The other processes should only run on what would be considered master servers.
So since it is just a matter of copying the binaries over and changing some addresses in the environment files under /etc/kubernetes, it
would be easy to tweak the procedure above to get a different infrastructure.