The standard rpiboot supports booting from a single "boot" directory. My version of rpiboot has been modified to support overlay boot directories based on the USB "path" the device is connected via. This allows the use of a custom configuration file for each slot on the Cluster HAT (or ZeroStem/USB Cable). This is used to give each Pi a custom cmdline.txt which sets the MAC address on both sides of the USB Gadget Ethernet device, mount the right NFS directory, etc.

I advise using a 16GB SD card as the image requires 5GB and MUST be expanded with "sudo raspi-config" on the controller before trying to boot any Pi Zero.

Issue with ClusterHAT-2017-07-05-lite-1-usbboot.zip - If you've downloaded the earlier version of this then you may find rpiboot gets stuck after booting the first Pi Zero. To fix this you need to update rpiboot by running the following command on the Controller Pi and then reboot.

sudo sh -c "cd /var/lib/clusterhat/usbboot/;git pull;make"

When using either OLD 2017-07-05 image with a Cluster HAT v2.0 you will also need to upgrade the tools to support the newer HAT - see the forum post for upgrade instructions.

Installation

The image should be written to an SD card for your Controller Raspberry Pi by following thestandard instructions.

Stretch based images have SSH disabled by default (on Controller/Pi Zeros) and the filesystem will automatically expand.

Older Jessie based image have SSH enabled by default on both the Controller and Pi Zeros, the Controller filesystem must be expanded on first boot using "sudo raspi-config".

The "pi" users password is set to "clusterhat" (without the quotes) you MUST change this as soon as you login AND then expand the Controller filesystem with "sudo raspi-config".

Also please remember to remove SD cards from the Pi Zeros :).

Network Overview

The Pi Zeros have an internal network bridged to "brint" on the Controller, they also have an external network bridged to "brext" on the Controller. The Pi Zeros use VLANs to separate these 2 networks. Untagged traffic is used for internal and VLAN 10 is used for external traffic. It is done this way arround to utilise mounting the NFSROOT filesystem from the kernel command line where configuring VLAN is not possible.

External Network

Usage [Cluster HAT]

The image as supplied expects the Cluster HAT to be plugged into thetop USB port closest to the Ethernet port (for 3B) and bottom USB port furthest from the Ethernet port (for 3B+) on the Controller Pi.

If using a Cluster HAT P1-P4 can be powered on using the standard "clusterhat on" command.

You should be able to watch the Pi Zero request files and boot from by looking at the screen session for "rpiboot".

Once connected it will show files being requested as it steps through the boot process and eventually after a few seconds the LED should illuminate on the Pi Zero and it will boot up. It may take more than a minute before the Pi Zero requests the external IP address and starts SSH but you should be able to ping 172.19.180.1/2/3/4 early in the boot process as this is used to mount the root filesystem over NFS.

Pi Zeros can be rebooted as normal and will be booted up again automatically.

Usage [ZeroStem / USB Cable]

This section is a bit long/more complicated than it needs to be right now, will be cleaned up - if you need help post on the forum (If you can post the output of "lsusb -t" before and after you plug in the Pi Zero it should be easy to spot even when using standard Raspbian).

The root file system for the Pi Zeros is stored in /var/lib/clusterhat/nfs/pX where X is 1-4. The base "/boot" filesystem is stored in /var/lib/clusterhat/boot/ and within this directory are symlinks to the specific root filesystems for each USB path.

These relate to the USB path names so in the example above this relates to.

1-1.2- Is the top left USB port on the Raspberry Pi 3 which is connected to the USB hub on the Cluster HAT.1-1.2.4 - Is the first USB port on the Cluster HAT which is used for P1.1-1.2.3 - Is the second USB port on the Cluster HAT which is used for P2.1-1.2.2 - Is the third USB port on the Cluster HAT which is used for P3.1-1.2.1 - Is the fourth USB port on the Cluster HAT which is used for P4.

To use a different USB path topology you'll need to alter these symlinks, first find out the USB path name where you're going to be plugging in the Pi Zero.

So if you're plugging in a Pi Zero with a USB cable or Zero Stem directly into...

Top Left USB port on a Pi 3 the USB pathname would be "1-1.2"Top Right USB port on a Pi 3 the USB pathname would be "1-1.4"Bottom Left USB port on a Pi 3 the USB pathname would be "1-1.3"Bottom Right USB port on a Pi 3 the USB pathname would be "1-1.5"

The 1-1.2.4 relates to "Bus 01" - "Port 1" . "Port 2" . "Port 4" (which is Dev 49 in the example above) so by plugging something into the port and running "lsusb -t" before/after you should be able to work out the path name you need.

So for example to change /var/lib/clusterhat/nfs/p1 to be the filesystem for a Pi zero plugged into the bottom right USB port on a Pi 3 you would need to update the symlink to "1-1.5".

If you have any problem working out the path name you should be using please ask on the forum.

Hacks

The rpiboot process runs in a screen session (under the root user) rather than a systemd service. I do this to make it easier to monitor what's going on when needed without having multiple lines per second end up in the log file. Once things stabalise I expect this to move to a systemd service. This is started in the /etc/rc.local file.

There are still issues with the rpcbind/nfs services not starting on boot so again these are restarted in the /etc/rc.local file which seemed to work OK in my testing.

raspi-config doesn't work as it tries to detect the "/boot" partition is a mountpoint which is isn't as it comes with the root filesystem you can appease it by mounting the "/boot" directory again on the Pi Zero.

sudo mount 172.19.180.254:/var/lib/clusterhat/nfs/p1/boot /boot

Replacing the "p1" as appropriate, you should then be able to run raspi-config to enable the camera/i2c/etc.

Problems/Questions?

If you have any problems, questions or suggestions about the images please use theforumto discuss it.

How?

If you want to make your own image or see how I created this image the script is available here but be aware the script has bugs, it expects to be ran with a ClusterHAT image as the source file on a Raspberry Pi with plenty of storage.

]]>https://8086.support/content/23/88/en/guide-to-using-the-rpiboot-test-image-on-the-cluster-hat_zero-stem-or-just-a-usb-cable.html
https://8086.support/content/23/88/en/guide-to-using-the-rpiboot-test-image-on-the-cluster-hat_zero-stem-or-just-a-usb-cable.htmlFri, 26 Oct 18 13:00:00 +0200How do I show disk usage breakdown including hidden directories?Using "du" with the following options will show usage of all files and directories including hidden (those starting with a dot).

du -sch .[!.]* *

]]>https://8086.support/content/1/92/en/how-do-i-show-disk-usage-breakdown-including-hidden-directories.html
https://8086.support/content/1/92/en/how-do-i-show-disk-usage-breakdown-including-hidden-directories.htmlThu, 30 Aug 18 21:16:00 +0200How do I setup CentOS 7 for KVM Virtualisation with a bridged network [OVH/Hetzner]?Coming from a minimal install the first step is to install the required software

Replacing the name of your network device create a backup of the current network configuration (this must be a PREFIX otherwise it will be treated as a network device and will be configured on boot causing issues).

Replacing the IPADDR/NETMASK with the inet initially shown on the network device, the GATEWAY IP from the output of the "ip route" command.

Initially I would try restarting the network without a reboot

# service network restart

If the network has been configured correctly the command prompt should return. If the command prompt doesn't come back you will need to boot the server in recovery mode and then either revert the changes from the bak file or correct any errors.

I would also advise doing reboot to ensure the network will start up like this as this is the basis of the network used later.

# reboot

Again if the server comes back after the reboot all is good, otherwise you will need to boot into recovery and resolve the issue.

Setup libvirt and configure the bridge

Start and enable libvirtd to start at system boot.

# systemctl start libvirtd
# systemctl enable libvirtd

Again we now need to copy the network file (changing the filename to your device as required) to make setting up the bridge network configration a little easier.

Comment out the HWADDR, TYPE, NAME, DNS1/DNS2/DNS3, IPADDR, NETMASK and GATEWAY and add the BRIDGE line which should leave you with something like this (we are calling the bridge to use with KVM "br0").

You will now need to add the commented out GATEWAY line to your network configuration

# vi /etc/sysconfig/network

And add the removed GATEWAY line

GATEWAY=10.10.10.1

Once these fines have been updated and checked restart the network service

# service network restart

Now check your new bridge has the correct configuration with the following commans.

# ifconfig br0
# ifconfig enp6s0

The br0 device should now have the IP address and the enp6s0 device should have no IP address associated with it.

Once you are happy with the configuration reboot the server.

# reboot

If everything looks OK after reboot you can now proceed with your KVM system and setup new Virtual Servers (with virt-manager for example). If your server doesn't come back after a few minutes you will need to boot into recovery mode and check the configuration files.

]]>https://8086.support/content/12/71/en/how-do-i-setup-centos-7-for-kvm-virtualisation-with-a-bridged-network-ovh_hetzner.html
https://8086.support/content/12/71/en/how-do-i-setup-centos-7-for-kvm-virtualisation-with-a-bridged-network-ovh_hetzner.htmlSat, 05 May 18 19:13:00 +0200How do I enable autologin on TTY running on the Controllers serial port?To enable auto logins on the TTY running on any serial port on the Controller log in and run the following command and reboot.

The following is a first attempt rough guide to setting up the Cluster HAT + Pi Zeros to boot using usbboot meaning no SD card is needed in the Pi Zeros in the Cluster HAT.

It uses about 5.7GB of disk space for 4 NFS roots (~800MB each root) but I'd advise using at least a 16GB SD card to allow some room for the filesystems to grow.

There are issues with this setup :( - see the bottom of this guide for a list.

The USB path (1-1.2.4) assumes you're connecting the Cluster HAT using the top left USB connector (looking at the end with the connectors) if this isn't the case your path will be different.

How does it work?

After powering on the Pi Zeros (without SD cards) they boot up and show up as a device. This is then picked up by a modified rpiboot (see "Modifications to rpiboot" below) which sends over the required files for the /boot directory.The root filesystem is then mounted using NFS from the controller over the Internal network.On the Pi Zero the network device usb0.10 (VLAN 10) is bridged to eth0 on the controller and obtains an IP address using the DHCP server on the network (see "Network Configuration" below)

Modifications to rpiboot

I have modified rpiboot to support "overlay" files based on the USB device path, this allows individual config.txt/cmdline.txt/etc. files to be sent to specific Pi Zeros using a single rpiboot process.

The new mode is enabled using the "-o" command line option and for example when used with "-d /boot" would first try to load files from /boot/USBPATH/ and if not found fallback to /boot/ . When using the Cluster HAT the USBPATH for P1 is "1-1.2.4", P2 is "1-1.2.3", P3 is "1-1.2.2" and P4 is "1-1.2.1" so when P1 is booting and looking for the config.txt file it will try to send /boot/1-1.2.4/config.txt but if the file isn't found will send /boot/config.txt as normal. This allows the use of a custom config.txt/cmdline.txt for each Pi Zero.

The network now uses VLANs internally to separate the Internal and External networks.

Your local network doesn't need to support VLANs these are all handled within the Controller/Pi Zeros. With the configuration described here the Pi Zeros will still obtain an IP address from the DHCP server on your local network.

The original br0 interface which previously bridged eth0 (Controller) and ethpiX (Pi Zero) is removed and replaced by the Internal "brint" and External "brext" interfaces.

brint [Controller] is configured with the static IP address 172.19.180.254 and is bridged to the (untagged) ethpiX interface (usb0 on the Pi Zero size).

brext [Controller] is configured to obtain an IP address from the DHCP server on the local network (as br0 previously) and is bridged to eth0 on the Controller and the VLAN 10 interface ethpiX.10.

usb0 [Pi Zero] the untagged interface is used for NFSROOT (as initrd doesn't support VLAN). A static IP address 172.19.180.X is set for the Pi (so 172.19.180.1 for P1, ..180.2 for P2, etc). This is used for the NFSROOT but you can also use it for SSH/etc. access.

usb0.10 [Pi Zero] VLAN 10 is configured by dhcp (over brext to the eth0 interface on the controller) and will pickup an IP from the local network.

If you're using multiple Cluster HATs on the same network you just need to reconfigure the Controller by changing it's hostname (to controller1/controller2/etc.) and to prevent MAC address collisions the Pi Zeros need to be switched to different MAC addresses (and hostname changed to p11-p14/p21-p24/etc.). The internal 172.19.180.0/24 IP addresses will not conflict as they're isolated from the external traffic.

I login using serial console on the Controller Pi (if you're using SSH you'll need to enable it first).

Login and do the normal first boot setup - set a ner password for the pi user, enable SSH, install the packages you'll need, download the Controller image (or copy it from your local machine) and expand the root filesystem.

You should now be able to use the normal command to power up the Pi Zeros (with no SD cards) using.

clusterhat on

They will take quite a while to boot but you should be able to see some progress from "brctl show" and "ifconfig" when the interfaces are added and the internal 172.19.180.x IPs should start to ping after the initramfs is running.

Known Issues

The main issue with this setup is that after a couple of reboots the Pi Zeros stop showing up correctly on the Controller as USB devices. The work around for this is to disconnect/reconnect the USB cable to the ClusterHAT.

After rebooting the Controller Pi the NFS related services need to be restarted.