where I write about building software, mostly

ScaleIO is a software-defined SAN product from EMC Corporation. It allows you to create a block storage cluster using commodity hardware. ScaleIO is a closed source product. It’s free for non-production use, for an unlimited time, without capacity restrictions.

Today I’m trialling ScaleIO on my homelab cluster to provide persistent storage for application containers. While I’m using Kubernetes to abstract compute, a product like ScaleIO allows me to abstract storage. The end result is that stateful applications can come and go, and it doesn’t matter which Kubernetes node they end up on. They will always be able to get access to their provisioned storage. Kubernetes enables this through Persistent Volumes, and the ScaleIO volume plugin is supported out of the box.

Node Preparation

My cluster nodes are running Ubuntu Xenial on bare metal. First, we’ll enable root login via ssh, so that the ScaleIO Installation Manager can log into each node and do its thing. On each node:

Note that each ScaleIO Data Server (SDS) device needs a minimum of 90GB. I had set aside 75GB on each node for this exercise and ended up having to resize a bunch of partitions. The GParted Live USB came in handy for that.

Gateway and Installation Manager Setup

Next, we’ll install the ScaleIO Gateway and Installation Manager. We only need to set this up on one of the nodes.

The Gateway requires a Java 8 runtime as well as binutils:

sudo apt-get install openjdk-8-jre binutils

The Gateway and Installation Manager are installed using a .deb file that was included in the ScaleIO download:

Cluster Install via the Installation Manager

Once dpkg is done, we can access the Installation Manager UI by pointing our browser to the server on which we have just performed the installation. We need to connect via HTTPS, and accept the certificate when prompted.

From then on it’s just a matter of using the fairly self-explanatory UI.

Add SDS Devices

After the Installation Manager has finished setting up all the nodes, it’s time to add some storage to the cluster.

Log into the Meta Data Manager (MDM)…

scli --login --username admin --password somepass

And add some devices. In the following example I’m adding an empty partition /dev/sda1 from the 192.168.1.101 node to the storage pool.

Ansible is an open-source tool that allows you to automate server provisioning, manage configuration and deploy applications.

Where does a tool like Ansible fit in today’s immutable infrastructure world? While containers are better at enforcing immutability, if I’m starting from bare metal, I still need a tool to bootstrap and manage the compute and storage clusters that my containerised workloads will use. That’s where Ansible comes in.

Installation

First, let’s install Ansible on our control machine. In my case that’s my development laptop. On macOS we can use Homebrew:

brew install ansible

We also need to install Ansible on the nodes that we’ll be managing. It looks like this on Ubuntu:

Ansible modules are reusable scripts that can be used via the ansible command and in Ansible Playbooks.

Next Steps - Playbooks

While using Ansible to run ad hoc commands against managed nodes is useful, its real power is unlocked via playbooks. Playbooks are Ansible’s configuration, deployment, and coordination language. They are written using YAML. Here’s an example from Ansible’s documentation website:

I’ve just finished the latest addition to the home office. I call it my mini Kubernetes tower, a five node cluster built out of Nano ITX boards.

What is It for?

Software development. These days I deploy the server side applications that I build to Kubernetes. This will be my home development cluster.

What Is It Made of?

The cluster has a total of 16 physical cores, 40GB of memory and 720GB of SSD storage. Three of the nodes are passively cooled quad-core J1900 Celerons. The other two nodes have dual-core i3 processors.

The bottom layer contains an 8-port gigabit switch. On top of that is the first node: An SSD mounted on a piece of acrylic, a few mm of clearance, then the mainboard. The other nodes follow the same pattern.

The tower is 25cm tall and has a square footprint of 12.5 by 12.5cm.

The Build

I had some time to plan the build while waiting for all the parts to arrive. I decided to use acrylic sheets to mount the network switch and SSDs on, and threaded rods to hold everything together.

The hardest part of the build was figuring out how to drill close to the edges of acrylic sheets without causing breakages.

The trick was to use a very slow drilling speed, and to stop just before the drill bit poked through. I then turned the sheet over and drilled through the other side.

I took a 1m M3 threaded rod and cut it into 25cm pieces. I placed a spacer on each side of where I was going to cut, performed the cut and filed the ends. I then unwound the spacers off to rethread the ends.

I started with the top cover.

Then added the motherboard for the first node.

The SSD was next.

And the first node was done.

I had already installed Ubuntu 16.04 on each node before hand, but I made sure to test each node as the build progressed.

It would have been inconvenient to have to take a faulty part out from the middle of the stack.

I mounted the switch last.

And then crimped a bunch of cables and networked everything together.

I powered all the nodes up and finally, success: We have blinking lights!

Then I extended the official Cassandra image with the addition of dnsutils (for the dig command) and a custom entrypoint that configures seed nodes for the container. The new entrypoint script is pretty straight forward:

I recently wanted to set up a local Kubernetes cluster for development on my Mac. I tried many howtos in various state of staleness and brokenness. The official Vagrant setup didn’t even work for me on OS X until yesterday.