Swarming Raspberry Pi – Part 1

Here’s the first in a series of building a robust docker swarm of raspberry pi hosts. This installment goes through installing the Pi(s), modifiying the Docker Daemon config, and starting up a test to verify that everyone can join the cluster.

The cluster which I am using consists of

Raspberry Pi B+ x 5

Raspberry Pi 2 B x 5

In order to verify that the cluster is working properly, there should be ten (10) members, five (5) of which have 4 cores.

Note: The image resizes itself to fit the card upon the first boot and then reboots. This takes a little bit (minutes at most). Just don’t be surprised when it reboots.

Hostnames

In order for this to work properly, the hosts need a unique name. It’s sometimes possible to configure DHCP to do this, but I went ahead and edited /etc/hostname myself since I wanted the most portability possible.

Like all good sysadmins, I am lazy. Doing the same thing over and over is boring. Besides, that’s what computers are for. So I wrote a script to make copies of the image and then edit the hostnames. Once that’s done I can then proceed to write the images to the SD cards.

The following script takes one argument, the location of the image you wish to modify — this should work for more than just Hypriot; it should work for just about any Rasberry Pi image. At present, it creates 10 instances and the NAME starts with apis-rpi- — you will likely want to change those. I may make the file a bit more generic at some point.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

#!/bin/bash

IMAGE=$1

DIR=${HOME}/images/pi-swarm

sudo umount/mnt2>/dev/null

mkdir-p$DIR

rm-f$DIR/*

foriin12345678910

do

echo"Copying #${i}"

NAME=`echo${i}|awk'{printf("apis-rpi-%02d\n",$0)}'`

FILE=${DIR}/${NAME}.img

cp$IMAGE$FILE

echo" Updating"

dev=`sudo kpartx-a-v$FILE|tail-1|awk'{print $3}'`

sudo mount/dev/mapper/$dev/mnt

sudo echo$NAME>/mnt/etc/hostname

sudo echo"127.0.1.1 $NAME">>/mnt/etc/hosts

sudo sync

sudo sync

sudo sync

sudo umount/mnt

sudo kpartx-d-v$FILE

echo" Done"

done

The Pi instances all need network connectivity which can be seen by the swarm manager; a switch works well. You can either physically assign IP addresses or use DHCP.

Updating Docker

In order for swarm instances to communicate, the docker daemon needs to be bound to a TCP port.

Docker should be running once again. I’ve left the unix socket enabled so that the “default” behaviour still works.

WARNING There are some security issues with this; a port is opened to anyone who can reach the host at which point they can start containers, etc.. Other than using this for play/testing on a private network, TLS needs to be configured.

To verify that the daemon is running with both the port and the socket, do docker info and docker -H 127.0.0.1:2375. You should see something like the following for both:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

$docker info

Containers:2

Images:22

Storage Driver:overlay

Backing Filesystem:extfs

Execution Driver:native-0.2

Kernel Version:3.18.8-hypriotos-v7+

Operating System:Raspbian GNU/Linux7(wheezy)

CPUs:4

Total Memory:925.3MiB

Name:apis-rpi-01

ID:LNUB:PB4W:KC6Z:JNHJ:LLH2:FYAY:HP4S:EWEJ:QYOQ:JRQT:XINL:4APW

Debug mode(server):true

Debug mode(client):false

Fds:20

Goroutines:22

EventsListeners:1

Init SHA1:6f77311608d545807a397d65468a964df6a37519

Init Path:/usr/lib/docker/dockerinit

Docker Root Dir:/var/lib/docker

Obtaining Swarm

I’ve created a Raspberry Pi docker container for swarm. To install it, you can simply pull nimblestratus/rpi-swarm. With more than a couple Raspberry Pis, this can be a bit tedious. clusterssh makes this a lot easier — it allows you to type in the same commands to multiple servers at once.

Discovery Service

In future installments I’ll be getting a different discovery service running, but for this test the default works.

1

2

TOKEN=`docker run--rm swarmc`

Starting the Nodes

On each node, execute the following: (this assumes that you’re using wired ethernet; for wireless you’ll need to change eth0 to most likely wlan0)

Hypriot is actually wheezy, just stripped down a bit with support for Docker baked in. Stock Debian doesn’t have support for Docker without some serious hacking at which point you’re better off starting with Hypriot and then adding packages you want. To compare, the Hypriot image is about 1G whereas the Raspbian is 4G. Admittedly the Hypriot folk are building with an emphasis on a light image focused on running containers, but since they’re based on Debian it’s just a matter of ‘apt-get’ing the packages you think are missing.

Thanks for this series, really helpful. Initially I wanted to use zettio/weave, the idea of having a meta network to which you can plug a device anytime and have it available is really interesting. Do you have any idea how weave would integrate into this swarm architecture you are putting together?

I’m interested in weave, too. Unfortunately as of the last I looked (the last week or so), someone reported on the Hypriot site that weave wasn’t working. I don’t remember the exact details, but I think it had something to do with the Docker port. Consequently, I’ve not given it a whole lot of thought.

I just did a little googling, however, and I see that there’s a number of folk who have used weave with the Pi. I’ll look at porting it this week and see what I get.

As far as integration, I think that a lot of it would depend on the discovery service used. Static addresses or a file containing a list of members would not work very well. Consul, etcd, or the Docker Hub token would likely work. An IP range might work with weave. I think that if you had a way of finding the “manager” node and assuming it was in the weave network then it should all work if you required that all of the addresses were weave addresses.

There is a possibility of bottlenecking, but that’s a potential issue with weave anyway.