ARMing Kubernetes with OpenEBS #1

Running stateful containers on Le Potato

Why not! It’s fun and extremely efficient! I know many people (including me) used to run ownCloud on their desktop computers. I have finally decided to retire my old desktop computer and was looking ways to keep my ownCloud instance alive and maybe even improve it a bit.

First, I ran ownCloud on GKE Kubernetes cluster and came to a conclusion quickly that it’s not what I needed:

I am used to the speed of USB 3.0 for large uploads when needed. I wanted to keep the option of (40MB+/sec) using USB. Which means, if I choose a budget ARM SoC route, then the board should have non-shared bandwidth for LAN and USB.

Low-power, possibly solar-powered. I’ll share my power consumption findings in the series of blog articles.

Everything on Kubernetes is more fun, right?

I was looking into Raspberry Pi 3 Model B option and after a quick trial realized that shared USB/Ethernet bandwidth and lack of MicroSD UHS support is not going to give me the performance I need and found the AML-S905X-CC Le Potato board.

Libre Computer Board, code name Le Potato, is designed as a drop in hardware replacement for the Raspberry Pi 3 Model B (In fact it has the exact same form factor and port locations) and offers faster performance, more memory, lower power, higher IO throughput, 4K capabilities, open market components, improved media acceleration and removal of the vendor locked-in interfaces.

This platform uses the latest technologies and is built upon proven long-term available chips. It is supported by upstream Linux and has a downstream development package based on Linux 4.9 LTS that offers ready-to-go 4K media decoding, 3D acceleration, and more.

Most importantly, Le Potato has almost double the performance of RPi 3 with 2GB memory and 50% faster CPU and GPU. It also has non-shared bandwidth for LAN and USB and MicroSD UHS support. I was able to get over 70MB/s read&write performance vs ~15-23MB/s on RPi 3. I also noticed that even under heavy load Le Potato has lower power consumption compared to Rpi 3. Redis benchmark below summarizes what to expect from the modern 28nm SoCs with faster DDR3/4 running ARMv8 kernel and userland.

It sounds too good to be true, right? Since Le Potato is new, I’ve decided to run both side-to-side and publish my experience.

In this blog post, I will focus on setting up a Kubernetes on a Le Potato Clusters, install ownCloud on top, and compare it to Rpi 3.

Software components used

I will start with Le Potato and compare against Raspberry Pi 3 on my next blog.

Flash Le Potato Armbian image on SD Cards

Armbian provides Debian and Ubuntu based builds for ARM development boards. Armbian Ubuntu is pretty much same as Ubuntu, except the desktop interface. Armbian uses the Xfce desktop environment, which is a lighter than Gnome or KDE. Its main advantage is its speed, and it’s ideal for systems with 256 MB to 512 MB of RAM. And, I plan to disable desktop anyways.

Server:
Engine:
Version: 18.02.0-ce
API version: 1.36(minimum version 1.12)
Go version: go1.9.3
Git commit: fc4de44
Built: Wed Feb 721:09:572018
OS/Arch: linux/arm64
Experimental: false
If you would like to use Docker as a non-root user, you should now consider
adding your user to the "docker" group with something like:

sudo usermod -aG docker murat

Remember that you will have to log out and back infor this to take effect!

WARNING: Adding a user to the "docker" group will grant the ability to run
containers which can be used to obtain root privileges on the
docker host.
Refer to https://docs.docker.com/engine/security/security/#docker-daemon-attack-surfaceformore information.

Note: If kube-dns is stuck in the Pending state. Follow the steps below to fix it and re init your master. This issue and the solution was mentioned here.

kubeadm reset

sudonano/etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Remove the $KUBELET_NETWORK_ARGS entry from the ExecStart, save the file, and reload systemd and kube services.

systemctl daemon-reload
systemctl restart kubelet.service

Initialize your master K8s node again.

Join Kubernetes nodes to the cluster

You can now join any number of nodes by running the command with the token generated during the K8s master initialization:

[email protected]:~$ kubeadm join--token 17c6f2.bd9fa915e6a2fcfb 10.10.0.131:6443--discovery-token-ca-cert-hash sha256:b4995d14fc8995d5ac271e49772b1cf5aa9fee48fa2729fd4ca7fefbbb0564ac[preflight] Running pre-flight checks.[preflight] Some fatal errors occurred:[ERROR IsPrivilegedUser]: user is not running as root[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`[email protected]:~$ sudo kubeadm join--token 17c6f2.bd9fa915e6a2fcfb 10.10.0.131:6443--discovery-token-ca-cert-hash sha256:b4995d14fc8995d5ac271e49772b1cf5aa9fee48fa2729fd4ca7fefbbb0564ac[preflight] Running pre-flight checks.[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.03.0-ce. Max validated version: 17.03[discovery] Trying to connect to API Server "10.10.0.131:6443"[discovery] Created cluster-info discovery client, requesting info from "https://10.10.0.131:6443"[discovery] Requesting info from "https://10.10.0.131:6443" again to validate TLS against the pinned public key[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.10.0.131:6443"[discovery] Successfully established connection with API Server "10.10.0.131:6443"

This node has joined the cluster:* Certificate signing request was sent to master and a response
was received.* The Kubelet was informed of the new secure connection details.

Run [cci]kubectl get nodes[\cci] on the master to see this node join the cluster.

If you forgot the cluster token, you can generate a new one with the command:

kubeadm token generate

Repeat all the steps above on all your nodes.

Install OpenEBS on ARM (Le Potato)

Similar to most of the ARM based hobby boards, Le Potato doesn’t provide any additional redundancy. Even using a RAID protected external USB device wouldn’t give me protection against node failure unless it’s some form of a shared network storage. They are both way over my affordability requirement. All I need is a replicated block device, so my container can survive a node or USB device failures.

OpenEBS provides a great solution for modern x64 architecture but currently doesn’t have a build for arm64 (armv8) architecture. Therefore, I’ve opened an issue here and started working on it myself. I did successfully build OpenEBS images for arm64 architecture from the repo base on the 0.5.3 release and uploaded custom images to my personal docker registry here. So, it is work in progress and please use it at your own risk, until it’s merged.

Next – Installing containerized OwnCloud on OpenEBS

Finding right container images to run on arm64 architecture is challenging. On the next article, I will build an OwnCloud container image running on Postgres database and both containers will store their data on OpenEBS persistent volumes.

My final goal is to build a mobile OwnCloud cluster installed in my family van, where storage is replicated to another cluster in my home lab.

6 Comments

tkaiser

1 year ago

me

•

1 year ago

Thank you for the comment. You are right. I will experiment with zram, since these devices have limited memory. With the current config swappiness is configured as follows:[email protected]:~$# sysctl -A 2>&1 | grep swap
vm.swappiness = 60
There is a good discussion on Armbian forum: https://forum.armbian.com/topic/5565-zram-vs-swap/

Da Xue

•

1 year ago

zswap is a better alternative if you are using large amounts of memory although the system still may crash if the IO gets too overloaded. zswap does not put the compressed pages into swap. instead, it decompresses it before sending it to swap.
zram is totally in memory will not offload to disk so it will OOM. compcache never made it into the kernel.

me

•

1 year ago

I’ve removed that requirement, looks like swapoff is enough to clean the “[ERROR Swap]: running with swap on is not supported. Please disable swap” message.
As an alternative during init “–ignore-preflight-errors” parameter can be used to skip all checks.