Steps

Sharded MongoDB on K8s with persistent volumes on NFS share

Step1 of 8

Step 1 - Deploy a kubernetes cluster

In this scenario, We are going to deploy a Sharded MongoDB with two shards.
To add a bit real-life conditions, We are going to use two nodes with kubernetes cluster on it.
Next, We will configure persistent volume claims for each shard and config database using nfs-client-provisioner.
Also, We will try to achieve some minimal pod affinity for our two nodes.

Task

For this scenario, We are using a shell script which deploys for us k8s cluster:/opt/launch-kubeadm.sh - This command is now running on HOST1. Please wait until it endsIf You are wondering what does this script do, please take a look at this tutorial.

You can now join the second host to the cluster by running the following command:
kubeadm join [[HOST1_IP]]:6443 --token 96771a.f608976060d16396 --discovery-token-unsafe-skip-ca-verification

Allow scheduling of pods on Kubernetes master node:kubectl taint node master node-role.kubernetes.io/master:NoSchedule-
After about a one minute both nodes should be Ready, You can check the statuses by executing this command on master node:kubectl get nodes
You do not have to wait, in the next step, We need to install NFS packages so it may proceed in the background.

Step 2 - Install'n'configure NFS server

We need nfs server for persistent volume claims for our shards, so let's configure one on host2

Task

Update and install nfs-kernel-server on host2:apt-get install nfs-kernel-server
and nfs client on host1:apt-get install nfs-common
then prepare shared directory (host2):mkdir /root/nfs-share
define what we want to share and who may connect(host2):echo "/root/nfs-share [[HOST1_IP]]/16(rw,no_root_squash)" >> /etc/exports

service nfs-kernel-server restart

Tests

Our NFS server is up'n'ready, let's test it:
Prepare mount point on host1:mkdir /root/host2mount -t nfs -o proto=tcp,port=2049 [[HOST2_IP]]:/root/nfs-share /root/host2
then You may check if files are accessible on both hosts:touch /root/nfs-share/testls /root/host2/

Step 3 - Install'n'configure nfs-client-provisioner

This chart installs custom storage class into a Kubernetes cluster using the Helm, package manager. It also installs the NFS client provisioner into the cluster which dynamically creates persistent volumes from single NFS share.

If You want to know more about Helm - package manager or nfs-client-provisioner this should be the best place to start

Task

Run below commands for init helm:helm init
extend its permissions so that Helm can deploy for us whole nfs provisioner:kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
then install and configure stable nfs-client-provisioner package:helm install --set nfs.server=[[HOST2_IP]] --set nfs.path=/root/nfs-share --name nfs-provisioner stable/nfs-client-provisioner
If You received Error: could not find a ready tiller pod it is because helm init does not finish yet, try again in few seconds.

Check if all is ready:kubectl get nodeskubectl get pods

Step 4 - Prepare to MongoDB deploy

We have a solid basement for MongoDB:

Kubernetes cluster

NFS server

nfs-client-provisioner

Now it is time for Mongo, but before that, You should take a look at how many things We have running now.kubectl get all --all-namespaces
Create a separate namespace for MongoDB to have things organised

Task

We will have a few yaml templates in a moment, so it is a good idea to create new working directory:mkdir /root/mongodb; cd /root/mongodb
Now We want to create a new namespace for our MongoDB deployment:echo "
apiVersion: v1
kind: Namespace
metadata:
name: mongo
" > namespace.yaml
From now We will use k except for kubectl to make things faster:k apply -f namespace.yaml
Now We can check if our new namespace for mongo only was created and what is inside:k get nsk get all -n mongo

Step 5 - MongoDB config server

Config servers store the metadata for a sharded cluster.
The metadata reflects state and organisation for all data and components within the sharded cluster.
The metadata includes the list of chunks on every shard and the ranges that define the chunks.

Step 6 - MongoDB shards

A shard contains a subset of sharded data for a sharded cluster.
Together, the cluster’s shards hold the entire data set for the cluster.

Description

Open shard1.yaml:cat shard1.yaml
This file and second one shard2.yaml have common body with a previous one - configsvr.yaml.
We create service for both shards, then the StatefulSet with three replicas, then launch mongod, but this time with the --shardsvr parameter, and listen on port 27018. In the end We create and attach persistent volume claims to each pod.
Let's try to put each shard on differ node: shard1 = masternodeSelector:
kubernetes.io/hostname : master
and shard2 = node01:nodeSelector:
kubernetes.io/hostname : node01

Task

Step 7 - MongoS

MongoDB mongos instances route queries and writes operations to shards in a sharded cluster.
mongos provide the only interface to a sharded cluster from the perspective of applications.
Applications never connect or communicate directly with the shards.

Description

Open mongos.yaml:cat mongos.yaml
This config is less complicated than any previous, it does not need persistent volume because all data is cached into ram.
We create service for mongos, then the StatefulSet with two replicas, then launch mongos, with addresses of MongoDB config servers replicaset, and listen on port 27017. And in this case, pods with mongos instances should be propagated on each node, same as configsvr.