Category Archives: Uncategorized

So, my MythTV box was starting to fill up. It had three 3TB drives in it. I also had three 3TB drives in my main desktop machine to hold a backup of the Myth box. With space running low, and with the cases pretty full of hard drives, it was time to do something.

I decided I would build a NAS, and buy some 6TB drives. My reasoning was that I could stripe pairs of 3TB drives together into logical 6TB drives. I would then have a Myth box with three 6TB drives, and a NAS with six 3TB drives and one 6TB drive. The NAS drives would be set up to look like four 6TB drives, and I could do a RAIDZ1 on those.

Over time, as the 3TB drives failed, I would buy 6 TB drives when necessary, and pair together the survivors of striped pairs that had failed. Eventually, I would have four 6TB drives in the box.

One reason for having a NAS is that I could just pick up the whole box, carry it to a relative’s house, and leave it there if I was going on a vacation. I don’t like having all my backups in the same place, even if important files are backed up in a fireproof/waterproof safe.

I do have sensitive files, so it was also important that the contents of the NAS be unavailable when it was booted up, until a passphrase is supplied.

It turns out you can’t do that, but the suggestion I received there, to partition the 6TB drive into two 3TB logical units and then put everything together as RAIDZ2 was a workable alternative.

So, here’s the procedure I worked out. My 3TB drives are on ada0, ada1, ada2, ada5, ada6, and ada7. My 6TB drive is on ada8. A lot of this had to be done on the command line. As I was eventually to figure out, there’s also a lot of stuff that I traditionally do on the command line that can’t be done that way anymore.

First, I create an encryption key:

dd if=/dev/random of=MainPool.key bs=64 count=1

I uploaded this key to Google Drive, as an off-site backup. The passphrase that is used with the key means that the key isn’t particularly useful by itself.

Now, we have to partition the 6TB encrypted drive. Note that the size of the drive looks different on the encrypted than the bare device, I think the block sizes are different. So, I used this sequence of commands, with the argument on the third line being half of the size reported by the ‘show’ command on the second line:

Running glabel status allows me to identify the gptids of the partitions. That’s important because I don’t know whether the adaN identifiers change when drives are removed or based on boot-time hardware probing. So, gptids are UUID labels that we can use to identify the partitions unambiguously.

All of this happened on the command line, but we can now switch over to the GUI. In the GUI, go to “Storage”, and select “Import volume”. Choose the encrypted pool option. You’ll have to supply the key (MainPool.key), which has to be on the computer that is running your web browser, so you can upload it. You will then be asked for the passphrase, which you type in. The system ponders for a while, and then the pool appears.

Next, I created datasets for the backups of the computers in my house. The main computer is called Londo, so I created a Londo-BACKUP dataset. Underneath that, I created separate datasets for my root partition, my home partition, and my encrypted partition. The MythTV box is called “mythtv”, I created a MythTV-BACKUP dataset, and underneath that separate datasets for the non-media partitions, and one for each media partition. I turned off compression on the media partition datasets, as that wasn’t going to achieve anything on H.264 files. With this granularity of datasets, I can snapshot the root filesystem and the user files separately, and I can avoid snapshotting the MythTV media partitions, which see a lot of churn in multi-GB files.

We now move on to the configuration. This was more difficult than it had to be, mostly because by now, having set up the drives that way, I was primed to use the command line for things. I know how to configure ssh, rsync, and so on, and I made my changes, but nothing worked. Turns out that the FreeNAS system overwrites those configuration files when it wants to, so my changes were being silently discarded. Many of these settings have to be altered through the GUI, not from the shell.

After FreeNAS installation, the sshd is set up to refuse port forwarding requests. I wanted to use those for my rsync jobs. I would alter the /etc/ssh/sshd_config file, and the changes would do nothing. The file wasn’t modified. Turns out that there are two sshd_config files. The /etc/ssh directory, while present and populated, is unused. The actual location of the sshd files is /etc/local/ssh, and they are subject to being overwritten, so I used the GUI to turn on port forwarding.

I was getting very slow throughputs on ssh, about 1/8 of the wire speed. I confirmed that wget ran at the expected speed, and that two ssh sessions each got 1/8 of the wire speed, so I was CPU bound on the decryption of the data stream. That was a bit surprising, it’s been a while since I saw a CPU that couldn’t handle a vanilla ssh session at gigabit ethernet speeds. So, I checked to see what ciphers were supported on the FreeNAS sshd, and tested them for throughput. I settled on “aes128-gcm@openssh.com”, which allowed me to use half the wire speed. Good enough, though the initial backup would take over 40 hours, rather than just 20. I avoided that by backing up over three separate ssh channels, so I could go at full wire speed by backing up different datasets in parallel.

On to rsync. I like to have security inside my firewall, and don’t like the idea of backups being sent over unencrypted channels. I also don’t want the compromise of a single machine to endanger other machines if that’s at all avoidable. So, simple rsync over port 873 wasn’t what I was looking for. I also wanted the machines in the house to be able to decide if and when to perform a backup, rather than having the NAS pull backups. That way my scripts could prepare the backup and ensure that their filesystems are healthy before starting to write to the NAS. The obvious choice, then, is rsync tunneled over ssh.

First, I generated an rsync key:

ssh-keygen -b 521 -t ecdsa -N “” -C “Rsync access key” -f rsync_key

I copied the private key to all the machines needing to make backups, and I put the public key into the authorized_keys file:

The default umask in bash on the FreeNAS box is 0022, so you have to be careful with permissions. Make sure to set files in .ssh to 0400 or 0600, to ensure that they are not ignored.

This authorized_keys file does not allow the user of the key to execute any commands. They can open a connection, and they can forward ports. So, each machine on my network can use this same key to open an encrypted connection to the rsyncd port on the NAS.

Next, we have to set up rsyncd. This has to happen in the GUI. There’s a box labelled “auxiliary parameters”, you just copy everything in there. It opens in the global section, so put in the individual section headers, and you can append anything you like to the base rsyncd setup. Here’s mine:

EDIT #1 (2017-09-05): Originally, I had the secrets file at /usr/local/etc/rsync/secrets.txt, but it turns out that, on reboot, extra files in the /usr/local/etc/rsync directory are deleted, so my secrets file disappeared and my backups failed. I have moved it to the parent directory now.

EDIT #2 (2017-09-07): Turns out the /usr/local/etc directory isn’t safe either. After performing an upgrade, I lost the passwords file again. I have moved it to /root.

With the ‘address’ keyword in the global section, we restrict access to the rsyncd to localhost. That’s fine for us, we’ll be coming in through ssh, so our connections will appear to come from 127.0.0.1. Other accesses will be blocked. I use chroot and numeric IDs because I do not have common UIDs between machines in my home network anyway, so I don’t care to remap IDs on the NAS. I run as UID/GID zero so that the backup has permission to create any files and ownerships that are appropriate. There is a secrets file that contains the plaintext passwords needed to have access to each module. Mine looks like this:

The appropriate password is also copied into a file on each machine being backed up, I’ve chosen to put them in ~/.ssh/rsync-password. Only the password, not the username. Make sure the permissions are 0400 or 0600.

This says that if the root user on the MythTV box just types “ssh freenas-1”, that it will go to the correct machine, with ssh stream compression, using the rsync key, protocol 2, no terminal, using the cipher we identified as acceptably fast, and will open a local forward port on 4322 on the MythTV box that encrypts all traffic and sends it to port 873 on the NAS box.

What does this do? First, it verifies that all the media drives are mounted. I created directories called xfs-vol on each drive. If those directories are not all present, it means that at least one partition is not correctly mounted, and we don’t want to run a backup. If a power spike bounced the box while killing a drive, it would start up, but maybe /myth/tv3/ would be empty. I don’t want the backup procedure to run, and delete the entire /myth/tv3 backup.

Next, we create the ssh connection to the NAS and record the PID of the ssh. We wait a few seconds for the connection to complete.

We generate a mysql dump. Backing up the MySQL files is rarely a good strategy, the resulting files generally can’t be used. The mysql dump is an ASCII snapshot of the database at a given instant, and can be used to rebuild the database during a restore from backup.

Because I use the ‘-x’ switch in rsync, each partition has to be explicitly backed up, we don’t descend into mount points. The next 4 lines send 4 non-media partitions into a single backup directory and dataset on the NAS. The “mythbackup” user is the username in /usr/local/etc/rsync/secrets.txt on the NAS box, it need not exist in /etc/passwd on either box.

Next are the media partitions. They are mounted on /myth, /myth/tv2, and /myth/tv3. To avoid leading cruft, we chdir to each mount point and then send the backup to the appropriate subdirectory of the module. Once everything’s backed up, we kill the ssh tunnel.

That’s pretty well everything for now. I might write another little article soon about what I did with the pair of 250GB SSDs. They’re for swap, ZIL, and L2ARC (see this article), with about 120GB left over for a mirrored, unencrypted pool holding things like family photos that can be recovered even if the decryption key is lost.

I was able to find time over several nights to assemble a metal puzzle. In a Google+ posting, Linus Torvalds mentioned working on a Star Wars model, and I was curious to see what other models were available by Fascinations. I eventually ordered this:

Products from Amazon.ca

It sat, unopened, on my desk for a few months, but I finally tore open the packaging and started putting it together. After about 15 minutes, I realized I wasn’t going to be able to do it with just fingernails, I needed a proper tool, very fine needle-nosed pliers. So, I ordered those, and waited a few days.

Products from Amazon.ca

Some of the tabs were very hard to place, I needed to borrow a magnifying visor to get the tabs into the slots, but things went relatively smoothly. I broke three pieces. Two of the breaks aren’t noticeable, there were enough tabs to hold everything together. The last piece broke when I was trying to remove it from the metal sheets, you can see a gap near the front of the upper cupola on the model where some of the length of that piece broke off. Here’s the final version:

Overall, it was a fun experience. Not frustrating, but it did require patience.

So, turns out that the birth of a new baby does reduce the amount of time available to other activities. With the baby turning 2 years old soon, and starting to do things on her own, I might be able to get back to writing here again in a while.

So, what have I been up to? My stack of books to be read continues to grow much more quickly than I can possibly read them. Right now, I’m reading

Products from Amazon.ca

I’ve had to cut my list of authors to read down quite a bit recently, just to keep my stack of books from growing too much. I still prefer physical books to e-books, though there are many times I’ve wished that I could continue a book on my phone while waiting for something. I’m not willing to pay twice for the same book for that privilege, and the Amazon Unlimited books don’t seem to have much overlap with my reading list, so I haven’t signed up for that.

Meanwhile, Chin Yi and I managed to watch a movie this week,

Products from Amazon.ca

With the baby going to sleep around 10:00, and various chores to finish off after that, it can take a few nights to get through a movie.

I’m posting now mostly because I want to continue using my Amazon API account when running the tellico program to keep track of books and movies that I own. They changed their terms today, and so I had to sign up as an Associate, to put in advertising links to their products.

Now, there are just a few more minor details to fix up in the ncurses UI in programming mode. We have to display the contents correctly, allow inserts part-way through, and implement a special behaviour. In programming mode, the GOTO key, when followed by a dot and three numeric digits, caused the calculator to move to that step number instead of inserting a new program step. This is the only context in programming mode in which a keypress is not recorded, but is acted upon, so we’ve added special-case code in the engine to handle it.

We also need a way to exit programming mode. I’ve decided that the exit keypress, CTRL-D, will, in program mode, exit back to run-mode, rather than exiting the calculator entirely. A patch was put in for that.

The current code is checked into the git repository under the tag v2015-01-03.

Here’s a screen shot of the calculator in programming mode. We will be inserting additional lines before the “+” in this program.