Note: This post is part of a series. Each post builds on the previous ones. If you are just trying to add one thing to an existing system that was not built following this series, then I cannot promise that these instructions will work for you, although they probably will. If you’ve started from something other than a non-NOOBS Raspbian image, then you’ll probably need to adjust for that.

Please refer to the series Introduction for a list of all the different posts in the series.

Self-Promotion: I have recorded this series as a screencast for Pluralsight:
(http://www.pluralsight.com/courses/raspberry-pi-home-server)
If you have a Pluralsight subscription, please consider watching it. Reading the instructions is one thing, but watching it done demystifies the whole process.

Thank you!

Sync What?

SyncThing. It’s another cloud synchronization program, very much like Resilio Sync, which I covered earlier in the series. It’s a free and open-source project though, and offers most of the same functionality as Resilio Sync, although not as much a Resilio Sync with a Pro license.

I’m just putting it out there as an option, because it’s easily installed on the Raspberry Pi, and has clients available for just about any platform you’d care to run it on. I’m using it to mirror my public share to my CrashPi.

Note: Make sure you check out the Resilio Sync post in this series as well. Look at both options and decide which you like better. It seems that running both Resilio Sync and Syncthing at the same time works just fine, but you probably won’t need both.

Installing Syncthing

Syncthing has its own repository and signing key, similar to a few of the packages we’ve installed previously. Download and install the signing key like this.

wget -O - https://syncthing.net/release-key.txt | sudo apt-key add -

Here, we’re downloading the key file from Syncthing’s own site, and installing it by piping it directly into apt-get with the “add” keyword. This key is required in order for apt-get to install any packages that it was used to sign.

Next, add the Syncthing repository to APT’s list of known sources by creating a new source list. As with previous examples, I’ll create a separate file rather than just tacking onto the main sources.list file.

sudo nano /etc/apt/sources.list.d/syncthing.list

Paste in the following

deb http://apt.syncthing.net/ syncthing release

Exit nano, saving your changes (Ctrl-X, Y).

Tell apt-get to update its list of repositories, and then install Syncthing

sudo apt-get update
sudo apt-get install syncthing

Now we’re ready to start configuring Syncthing. Start Syncthing from the command line.

syncthing

Configuring from a Local Browser

If your Pi is hooked up to a monitor, and you’re running the desktop, then a browser window should open automatically, and in a few moments, the Syncthing web UI will appear.

This web UI is very full-featured, and you’ll use it to set up folders that you want to sync between devices. I’ll show an example of that a little later on. The only problem is that the UI is only available from a local browser running on the same machine as Syncthing. If you don’t mind remoting in to make changes, then you can just skip ahead, but if you’re like me, you’d like to connect directly from the browser on your regular computer.

This is easy enough through the web UI. Click the “Actions” drop-down in the upper-right of the page, and then “Settings”.

In the right-hand column, change the “GUI Listen Addresses” from “127.0.0.1:8384” to “0.0.0.0:8384”. This will allow Syncthing to be administered from other computers on your home network. We don’t want to leave it wide open to just anyone though, so fill in values for the “GUI Authentication User” and “GUI Authentication Password” as well. You can decide to make up an entirely different user name, or just use “pi” and its password. It’s up to you.

Configuring from a Remote Browser

If your Pi doesn’t boot to the desktop, or you’re doing this installation through a remote SSH session, then obviously the local browser is not going to be an option, but we can still make the required changes.

Now that Syncthing has been run once, it has created a config file where we can make the same adjustments. Press Ctrl-C in the terminal window to shut down Syncthing.

Change the “127.0.0.1:8384”, which refers to the local machine, in this case the Pi itself, to “0.0.0.0:8384”, which means any machine on the local network. Technically, you could forward a port through your firewall and make Syncthing accessible from outside your house as well, but there’s no reason to do that. If you set up a VPN as covered in my OpenVPN post, you can always connect securely to the Pi and make any changes you need from there.

Give Syncthing a moment to get started up, and then open a browser from your regular computer. Connect to the Pi, port 8384. The web UI will open up, and after a few moments you can expect to see a warning that the web UI is now configured to allow remote access, but isn’t going to require a password.

Click the “Settings” button, and fill in values for the “GUI Authentication User” and “GUI Authentication Password”. The same way as we did in the previous section. Click the Save button, and you’ll soon be prompter to re-authenticate with the name and password you just set up.

Sync some things

For a demonstration, I’ll be syncing a folder between two of my servers, my main Raspberry Pi Home Server, and my CrashPi (although sadly, CrashPlan isn’t working right now). By default, Syncthing already created a folder called “Sync” in the pi user’s home folder. Following the same directions above, I’ve installed Syncthing on my second server already, and it now has a similar default folder already created. All I need to do is introduce these two Syncthings to each other.

Click the Actions drop-down in the upper-right corner of the page again, and then “Show Id”. You’ll get a page with a QR code on it, as well as a text representation of this Syncthing’s “Device Id”. The QR code can be used by the mobile versions of Syncthing so that you can sync things to your phone. For now, copy the text above the QR code. Make sure you get the whole thing.

Switch to the Syncthing web UI running on the second device, and click the “Add Remote Device” button in the lower-left corner of the page. You may have to scroll a bit to see it. Paste the first Syncthing instance’s Device Id in the first field. Fill in a name so that you can keep track of which device is which, and then check the “Default Folder” checkbox at the bottom. Fincally, click “Save” to dismiss the dialog. There are a lot of other options here, but I’ll leave it up to you to explore Syncthing’s documentation on your own. Just click the “Help” button in the upper-right and you’re off.

Return to the web UI of the first Syncthing instance, and you should see a prompt asking you to confirm the addition of the second instance. This ensures that even if someone were to get ahold of your Device Id, they couldn’t just help themselves to your stuff. You need to give the okay first. Click “Add Device”, and you’ll see the same dialog that you were looking at a moment ago, but most of the fields will be filled in for you already. Check off the “Default Folder” checkbox again, and then click the Save button.

Finally, switch back to the other instance, and you’ll see another prompt asking you to confirm sharing the default folder.

Time to Test

Open a second Terminal window, SSH session, or simply use the file browser on the desktop to create a new file in the “Sync” folder that Syncthing created for you. Perhaps name it after the device on which it was created so we can keep track. Do the same on the other computer, and then observe what happens. If you’re lucky, the files will magically sync from one computer to the other. At the time I’m writing this, there’s some kind of bug when adding new folders, and you’ll need to go to the Actions drop-down and click “Restart” on both instances to get things flowing. After that though, the folders on each computer should stay in sync.

As with and file synchronization technology, it’s possible to have conflicts. If you edit the same file on both systems, or try to sync a file that receives regular changes from both places, things are going to get ugly. Rather than just clobbering what the other computer did, Syncthing will create “conflict” files, and leave it up to you to sort things out. This is good in that you won’t have one version simply overwriting another, but it does mean that you’ll have to be involved in fixing it when things go wrong.

Try to use Syncthing only for items that you want available everywhere, but will typically only make changes from one place at a time. Refer to Syncthing’s documentation for help with synchronization strategies.

Stopping Syncthing

Since we started Syncthing from the command line, that command line has been held up waiting for Syncthing to finish. You can press Ctrl-C in the terminal window to shut it down, or go to the Actions drop-down and click “Shutdown”. In either case, your command line will wake back up so you can issue more commands.

If you want to start Syncthing without locking up your command line, you can always append an ampersand (&) to the command to run it in the background.

syncthing &

Since you can’t hit Ctrl-C to stop it anymore, you’ll have to use the menu to shut down this instance if you want. Syncthing will also keep interjecting things into your terminal session, which can be annoying. This approach also means manually starting Syncthing every time you reboot, though.

Starting Syncthing automatically

Logging in and starting Syncthing manually isn’t going to work long-tern. Fortunately, Syncthing’s authors have done most of the hard work for you here. There’s already a service description set up and waiting for you. All you need to do is enable it and reboot.

sudo systemctl enable syncthing@pi.service
reboot

That’s it. When the pi reboots, Syncthing should be up and running as the “pi” user automatically.

At this point, I refer you to Syncthing’s documentation for more information on its configuration, care and feeding. It’s a very capable program, and I’ll be interested to watch its progress.

Troubleshooting

A couple days ago, I started having problems with my server. It would disappear from the network entirely. No SSH, no VPN, no VNC or even Samba. I had recently done a round of update/upgrade, so I figured something must be up with that. Looking at the logs, and the activity in htop, I saw a lot of entries about SyncThing, which I honestly haven’t really been using that much. I rebooted a few times in search of an answer, and noticed that after the server locked up on me again, SyncThing was responsible for the last entry in the syslog.

I don’t know that this means SyncThing is actually responsible, but disabling it made everything stable again. It could be that I simply had too many things running at once, and reducing the load made the difference. I’ll be keeping my eye on it, but for now, I thought it was worth tacking onto the post.

Note: This post is part of a series. Each post builds on the previous ones. If you are just trying to add one thing to an existing system that was not built following this series, then I cannot promise that these instructions will work for you, although they probably will. If you’ve started from something other than a non-NOOBS Raspbian image, then you’ll probably need to adjust for that.

Please refer to the series Introduction for a list of all the different posts in the series.

Self-Promotion: I have recorded this series as a screencast for Pluralsight:
(http://www.pluralsight.com/courses/raspberry-pi-home-server)
If you have a Pluralsight subscription, please consider watching it. Reading the instructions is one thing, but watching it done demystifies the whole process.

Thank you!

SSH is a simple way to remotely log in to your machine’s command-line interface, but that’s not always the most convenient way to work. There is a wonderful web administration system called Webmin that can handle a lot of the “magic” of system configuration. Webmin can take care of a lot of the tasks you’d normally do from the command line, but in a much friendlier way. In addition, you can add Webmin modules for many of the features we’ll be adding to the Raspberry Pi in this series.

Note that this is totally optional. Webmin doesn’t really give you anything that you can’t already get in other ways, and now that Raspbian images come with VNC remote desktop support built in, you can do pretty well without Webmin. It’s still a pretty convenient “dashboard” to check up on your Pi, though.

It’s pretty easy to install, so let’s get started.

You can’t just install Webmin through apt-get like the other software packages so far because apt-get doesn’t know about Webmin, or at least it doesn’t know about it yet. There are several approaches to a Webmin installation, but I’ve found that the easiest is to simply tell apt-get where to get the packages it needs for Webmin.

apt-get installs software based on a list of internet servers that it uses as sources to download from. You can edit the main apt-get source list, as I did in the previous version of this series, or you can add a new list specifically to support the one package you want to install. This has the advantage of keeping things separated rather than putting all your sources in one file together. This way, you know which sources were added to support which package.

Create a new list file just for webmin like this:

sudo nano /etc/apt/sources.list.d/webmin.list

When the editor appears, add the following line to the empty file:

deb http://download.webmin.com/download/repository sarge contrib

Note: The word “sarge” in the line above is the name of the distribution Webmin was created for. After Sarge was Wheezy, and then Jessie. Debian releases, and by extension Raspbian releases, are all named after Toy Story characters. Sarge is quite an old release at this point, but I guess the Webmin team haven’t needed anything that the Sarge release can’t provide, so they haven’t felt the need to move on yet. That doesn’t mean they’re not maintaining Webmin. At the time of this writing, the latest version in the Sarge repository was from October 3rd, 2016.

Press Ctrl-x,y,enter to exit nano, saving the file. Next you’ll need to import the signing key that verifies the packages coming from the new repository. These next few commands need to be run as the actual root user of the machine. This is the first time this series has done this, so I’ll break it down for you. Type the following to temporarily become the root user:

sudo su

Your command prompt will change, losing all of its color, and becoming more sinister, dark, and dangerous looking like this:

You are now operating as the root user of the machine. The root user can do pretty much anything. Unlike the Windows world, Linux users try to spend as little time in “God-mode” as possible.

Type the following commands to import the Webmin repository’s signing key:

That last “exit” tells the system that you want to stop being the root user now, and go back to being “pi”. The final result should look like this:

Now that you’ve added Webmin’s repository to the list of places apt-get will look for stuff, update the list of available packages again. According to commenter Kevin Liston,

sudo apt-get update

Now that apt-get knows where to get Webmin, you can install it using apt-get.

Note: You may need to install https support for apt get as well. I can’t confirm at this time because I already had it installed on all of my Pis anyway as part of other installations, but it is a good thing to have anyway, so I’ll include it here.

Once again, the “-y” will stop apt-get from asking us if we’re sure before continuing.

You can expect this installation to take a little while without providing a lot of feedback. Be patient. When the installation is complete, you can test it by opening a browser, and navigating to the IP address of your server, but specifying https and port 10000 rather than the default http port of 80. If you forget the “https” part, you’ll see a friendly page that offers to redirect you. Either way, you’ll probably also get a warning about the site not having a valid certificate, which of course it doesn’t. Since you own this server, though, it should be safe. Proceed to the page anyway, and you’ll see a login page like this:

Log in as “pi”, and you’ll see the main Webmin interface, which looks like this:

Poke around a bit, and see what Webmin is all about. You can monitor storage and memory usage from here, be notified about updates, apply them, manage user accounts, and a lot more. You can accomplish a lot of the same tasks over SSH, but Webmin can make things a bit more convenient, and is available from any computer on your home network without having to install an SSH client.

Look inside the “Un-used Modules” section to see all the things that Webmin could help you manage, if you had those packages installed. It’s a pretty impressive list.

Wrapping up

You’ve reached another milestone, and I strongly recommend politely shutting down the Raspberry Pi (sudo shutdown –h now), and taking another backup of the SD card or OS partition.

Note: This post is part of a series. Each post builds on the previous ones. If you are just trying to add one thing to an existing system that was not built following this series, then I cannot promise that these instructions will work for you, although they probably will. If you’ve started from something other than a non-NOOBS Raspbian image, then you’ll probably need to adjust for that.

Please refer to the series Introduction for a list of all the different posts in the series.

Self-Promotion: I have recorded this series as a screencast for Pluralsight:
(http://www.pluralsight.com/courses/raspberry-pi-home-server)
If you have a Pluralsight subscription, please consider watching it. Reading the instructions is one thing, but watching it done demystifies the whole process.

Thank you!

Important notes:

Of all the parts of this series, this is the one that seems to have the most frequent breaking changes. Every time I turn my back, CrashPlan updates itself and throws out all of the patching and tweaking that made it work on the Pi in the first place, and the most recent breaking changes are pretty terminal.

At the current time, there is no simple way to get things working properly on the Pi. There are some hacks to install an old version and prevent it from updating itself, but they have problems of their own. For instance, you’ll keep getting emails from Code42 insisting that nothing has been backed up in weeks or months, even though CrashPlan itself reports that things are going just fine.

There’s also the issue of version compatibility. The CrashPlan instance running on your desktop will keep on updating itself, and eventually the difference between the version on the Pi and the version on your desktop will be so far out of sync that they will probably refuse to talk to each other, so in this version of the server, I’m going to go with a workaround that should be more permanent, but has significant drawbacks over a full installation. At the very least, this will get you a working backup for safety while I continue to research a way to get the full system running again.

You can refer to the original CrashPlan article, and the enormous tail of comments it has grown if you want to try to install CrashPlan directly on the Pi. I’m going to sidestep that and not install anything on the Pi at all.

CrashPlan-less CrashPlan

The secret is that since v4.3, CrashPlan has grown the ability to be installed on a per-user basis. This will allow CrashPlan to back up to a network location, which it can’t do otherwise. There are good reasons for this, but we don’t need to go into them here.

This ability comes with a cost, though. Since CrashPlan won’t be running as a service in the background when no-one is logged in anymore, the computer won’t get backed up until you log back in. If you’re the only user of a particular computer, then this won’t really matter. Instead of logging out at the end of the day, just lock the computer, and everything should still work just fine.

Create the backup share

Create an entirely new share as described in the Samba post, or a “backups” folder inside your existing public share. I’m creating a new share for organizational purposes. Linux or Mac OS user will need to create a symbolic link to the share somewhere in their filesystem. Windows users will “Map” the share as a drive letter or location within NTFS.

Navigate to the backup share or folder in Windows Explorer (e.g. \\rphs\backups). Right-click the backup share you created above and then “Map Network Drive”. Choose an unused drive letter, and leave “Reconnect at sign-in” selected.

Note: This screenshot is from me setting up my CrashPi as a destination.

Per-user CrashPlan

Install or re-install CrashPlan on your main computer following Code42’s per-user instructions. Basically, uninstall CrashPlan if it’s already installed, and then re-install choosing “Onlyfor me” when prompted for an installation type by the installer.

You’ll need to go through the whole CrashPlan setup process again, signing in to your account, providing a password, etc. If you were backing up to an external hard drive, then you should “adopt” that backup to avoid starting over from scratch.

Once you’re past all of that, you can now add the new, mapped network location as a destination.

After this, it’s business as usual. When your computer is on the home network, CrashPlan will see the mapped drive and back up to it. If your computer is portable, then when you’re on another network the drive will simply not be available, and CrashPlan will stop backing up until the mapped drive is available again. There’s one little wrinkle, though. If CrashPlan notices that a folder destination has gone offline, it doesn’t seem to be very good at noticing when it comes back. This presents a problem for laptops, and may require you to restart before CrashPlan will notice the location and start using it.

It’s not quite as nice as having a real remote CrashPlan instance, but it will do for now, and won’t break when Code42 pushes out an update every week. Windows may complain that not all mapped drives could be reconnected when you sign in, and you may have to reboot for CrashPlan to see the network location, which can be kind of irritating. But it works, and, more importantly, will continue to work. It just takes a little more hand-holding. If you’re backing up from a computer that’s always on the home network like a desktop, then this won’t affect you.

I’ll keep evaluating this approach, and hope that a better solution comes along, but this’ll get you a working backup solution until then.

Note: This post is part of a series. Each post builds on the previous ones. If you are just trying to add one thing to an existing system that was not built following this series, then I cannot promise that these instructions will work for you, although they probably will. If you’ve started from something other than a non-NOOBS Raspbian image, then you’ll probably need to adjust for that.

Please refer to the series Introduction for a list of all the different posts in the series.

Self-Promotion: I have recorded this series as a screencast for Pluralsight:
(http://www.pluralsight.com/courses/raspberry-pi-home-server)
If you have a Pluralsight subscription, please consider watching it. Reading the instructions is one thing, but watching it done demystifies the whole process.

Thank you!

Sometimes you want to download something large. “Transmission” and “Deluge” are cross-platform BitTorrent servers that will run quite happily on a Raspberry Pi. They can be used to host, or “seed” a torrent for other users to download, or to download other people’s torrents for you, freeing up your main computer to do other jobs or to sleep.

What’s a BitTorrent?

Let’s say I wanted to download the current Raspbian image file. I could get it straight from the Raspberry Pi website, but that means that I’m competing for bandwidth with all of the other people trying to get that same file, and that bandwidth is costing the Raspberry Pi Foundation money, money they could be using to build more cool toys.

Torrents work by asking all of the computers downloading a particular file to talk to each other and trade the bits they already have amongst each other instead of everybody downloading the same bits of the same file from the Raspberry Pi Foundation’s server. Through the magic of BitTorrent, one computer could says to the others “Hey, I already have the first half of the file, does anyone have the second?”, and another computer might answer “Well, I don’t have the whole second half, but I have most of it. I’ll give you what I have if you give me the first third, ‘cause I’m totally missing that”.

At the center of it all is one computer that started the whole thing off by hosting the complete file first, and staying on to serve as “matchmaker” when new computers get added to the mix. In the case of the NOOBS software, that’s the Raspberry Pi Foundation’s server again. The foundation’s server started with a complete copy of the file, from which it created a .torrent file. The .torrent file describes the file you’re trying to get, providing information like how big it is, how many chunks it’s been broken into, and providing a checksum hash of the complete file so that you can be sure you got all the pieces right, and to stop jerks from ruining it for everyone by hosting files with the wrong bits in them.

You start a download of the large file you want by first downloading the much smaller .torrent file. You feed this file to a BitTorrent client like Transmission or Deluge, which contacts the original server to see what peers are online sharing the load. Then, the client starts getting bits from all the different peers in the network, and only contacting the original server for those parts that no-one else has yet.

When the download of the large file is complete, the client stays online as part of the network, and just keeps on sharing (or “seeding”) the bits of the large file to anyone else who might need them.

Torrents have a kind of a bad reputation since they have become a kind of ersatz peer-to-peer file sharing mechanism for trading things like Hollywood movies, but they can be terribly useful for anyone that wants to mitigate the bandwidth costs of distributing large files on the internet.

Transmission vs. Deluge

The first thing you’ll want to decide is whether you want to go with Transmission or Deluge. They both do the same job, but Transmission’s interface is much simpler. Transmission got a bad reputation of its own when its installer was found to be spreading malware twice in 2016. This apparently wasn’t a purposeful attack by the Transmission team, and they moved quickly to correct the situation, but some people are uncomfortable with Transmission even now.

There is an excellent guide to installing Deluge at HowToGeek, but for now, I’m sticking with Transmission for its pure ease of installation on the Pi. I may do a Deluge write-up later on, in which case I’ll update this post to refer to it, but for now I’m just going to show you how to install and configure Transmission.

I don’t recommend installing both at the same time. While their respective web front-ends use different ports, their background daemons will want to use the same port, and will interfere with each other. I’m sure you can configure your way around this, but you only need one anyway. Pick your favorite and install it.

Note: I have found Transmission to be one of the most destabilizing influences on the Pi. I have not yet given Deluge a full test, but with Transmission running, my Pi becomes very sluggish and unresponsive when downloading very large torrents. On average, this may not affect you, but I suspect it has everything to do with the size of the thing you’re downloading. I’m talking about multi-hundred gigabyte downloads here, much larger than your average download. I’m not sure Deluge would fare much better, but I intend to find out at some point. If you regularly download HUGE items, then you may want to dedicate a second Pi with its own hard drive to this task rather than your main home server.

Installing Transmission

To install the Transmission daemon (that’s a service to you Windows folks), type the following at the command prompt.

sudo apt-get install transmission-daemon

It’s a pretty simple installation, but will require some configuration before it’s ready to use.

Configuring Transmission

Before editing the configuration file, make sure the transmission daemon is stopped.

sudo service transmission-daemon stop

Transmission keeps its settings in a JSON-formatted text file. Use Nano to edit it.

sudo nano /etc/transmission-daemon/settings.json

Change the following settings, modifying the download path to match whatever you called your external drive, and changing the user name and password. I’m going to make mine match the pi user so it’s easier to remember. Don’t worry about the password being in plain text. After we restart the Transmission daemon, it will encrypt the setting so no-one will be able to read it.

Note: You could limit the computers that are allowed to see Transmission’s web interface by leaving “rpc-whitelist-enabled” set to true, and then entering a comma-delimited list of IP addresses. You could also set the whitelist to “192.168.*.*” to allow any computer on the local network to see the interface. Since I’m not exposing Transmission’s ports through my firewall, turning the whitelist off achieves pretty much the same effect though.

Exit Nano, saving your changes (ctrl-x,y,enter).

Once again, a program is managing its own list of passwords. You get that a lot in the Linux world. I’ve kept the default user name of “transmission” here, and used the default password for “pi”. Pick whatever you want. Finally, restart the transmission daemon (service).

sudo service transmission-daemon start

Transmission provides its own web interface, so there is nothing to add to Webmin. Open a browser and navigate to the Raspberry Pi, port 9091, and after logging in with the credentials you created in the config file, you should see the Transmission interface.

Time to give back

At this point, I’d like to suggest that you seed the archives for NOOBS and other Raspberry Pi images. Click on the open folder icon in the top left, enter the url of one of the torrent files, and click “Upload”.

The latest version of the NOOBS and Raspbian torrents are always available from the following addresses:

A row for each .torrent file you upload will be added to the list, and the torrent will begin downloading. Click on the new row to select it, and click the blue “i” icon in the upper right to get more detailed information about how the download is progressing. When all the bits have finished downloading, and a torrent goes into “seeding” mode, its progress bar will turn green like this:

You might consider seeding all of the Raspberry Pi images if you can spare the space. Remember to pay attention to how much CPU is being used, and don’t overtax your Pi. If you seed too many torrents, there may not be much CPU left for other tasks.

There are Transmission “remote control” applications available for many platforms that will give you greater control over the download process, but they are beyond the scope of this post. You can start by looking at the Add-Ons section of the Transmission website (https://www.transmissionbt.com/resources)

Wrapping up

Now that Transmission is installed and running, you should shut down the Pi and take a backup of the SD card or OS partition as usual.

Note: This post is part of a series. Each post builds on the previous ones. If you are just trying to add one thing to an existing system that was not built following this series, then I cannot promise that these instructions will work for you, although they probably will. If you’ve started from something other than a non-NOOBS Raspbian image, then you’ll probably need to adjust for that.

Please refer to the series Introduction for a list of all the different posts in the series.

Self-Promotion: I have recorded this series as a screencast for Pluralsight:
(http://www.pluralsight.com/courses/raspberry-pi-home-server)
If you have a Pluralsight subscription, please consider watching it. Reading the instructions is one thing, but watching it done demystifies the whole process.

Thank you!

Now that the Raspberry Pi Home Server is up and running off of the hard drive, we’re presented with a problem. We can’t just take a backup of the SD card anymore. In fact, what’s on the SD card probably shouldn’t change from this point on. All the important stuff is on the OS partition of the hard drive.

In this post, I’ll present a couple different methods for backing up the hard drive using a secondary computer, or just using the Pi itself and a second SD card.

Backing up from a Windows computer

I’ve been using and advocating HDD Raw Copy over Win32DiskImager in this second edition of the Raspberry Pi Home Server series because of its ability to compress the image files on the fly, and that ability can be put to good use here as well. All you need to do is hook the Pi’s hard drive up to your Windows computer, and use HDD Raw Copy to copy the drive to an image file just like you’ve been doing with the SD card. It’s exactly the same process. If you’re using Win32DiskImager, the process is the same, only the resulting image files will be the same size as the drive you’re backing up, which brings me to the downside of this method.

Both HDD Raw Copy and Win32DiskImager are going to want to back up the entire hard drive. Not just the OS partition, but the data partition as well. If you have the space to keep that kind of backup around, then good for you, you can restore the entire server, including the OS and Data partitions as well as all of their contents in one go. It’s going to be slow, and it’s going to take up space, but you can do it.

This is part of the reason I keep my actual shared stuff on a completely separate drive now. I like the separation between the devices that make up the Pi itself, and the stuff that the Pi is sharing. I only have to back up a 120GB drive, and it’s mostly empty space, so my compressed image files are not that large.

If you can build your setup like this, I highly recommend it. If not, then you’ll need to continue reading and decide which is right for you.

Backing up from the Pi itself

If, for some reason, you don’t have access to a full Windows, Mac, or Linux computer, you can back up your hard drive by using another Pi. If you’re stranded alone on a desert island and only have a single Pi with you, then you’ll need to do a little prep work, the first step of which will be establishing a power grid.

Since this is the worst-case scenario, we’ll get it out of the way now. We’re going to create another SD card specifically for making backups. The Raspbian Lite image will have everything we need on it, or you can use a full Raspbian installation too. Just prepare another SD card, exactly like we did in “Installing the OS“, and then proceed to the “Backing up from Linux” section.

Backing up from Mac OS

Mac OS is built on top of Unix, which Linux is a sort of clone of, so things work pretty much the same there. You’ll just need do the same thing as the Linux users in the next section.

Backing up from Linux

The process for backing up from Linux is exactly like what you did to create the SD card in the first place, except that you’ll be using dd to copy from the Pi’s OS partition to another spot in your filesystem. This could be somewhere on the hard drive of your main computer, but for this demo I’ll be creating an image of the OS partition on the Data partition of the same drive. Just adjust the file paths to match where you want the image and you’re set to go.

Shut down the Pi, and transfer the hard drive to your main computer, or swap SD cards to the new one you created above. If you’re doing this from a single Pi, then I would also take this opportunity to unplug any other drives you might be using, leaving nothing but the system drive attached, and then start up the Pi with the new SD card.

Use blkid to figure out where the Pi’s hard drive got attached (not mounted, mind you).

sudo blkid

In my case, the hard drive shows up at /dev/sda, and the partitions are /dev/sda1 and /dev/sda2, with sda1 being my OS partition, and sda2 being my data partition. If you’re using a separate Linux system, then sda is probably your own hard drive, and the Pi’s drive will be sdb, sdc, sdd, etc.

I’m writing my image to the second partition of the same hard drive, so I’ll need to mount it first so that I can write files to it. Since this is the first time I’m doing this, I’ll also need to create a placeholder directory for the drive to mount at.

sudo mkdir /mnt/data
sudo mount /dev/sda2 /mnt/data

Now, we’ll use the dd command to copy the sda1 partition to sda2 (or wherever you’ve decided to store the image). The dd command is part of “coreutils”, and newer versions have the ability to give you a progress report as they go. See which version you are running.

dd --version

If it says 8.24 or above, you’ll be able to get a status report with the “status=progress” switch. Back up the OS partition like this.

sudo dd bs=1M if=/dev/sd?? of=IMAGEPATH.img status=progress

Unfortunately, Raspbian is still on 8.23 at the time I’m writing this, so status=progress won’t work. If your OS partition is still relatively small, then doing without feedback is a perfectly viable option, so you can proceed like this.

sudo dd bs=1M if=/dev/sd?? of=IMAGEPATH.img

If your OS partition is very large, or you opted to have just one partition, then you’ll probably want some kind of feedback. Install the dcfldd package through apt-get.

sudo apt-get install dcfldd

The syntax for dcfldd is very similar to that of dd, but it will report its progress as it goes.

sudo dcfldd bs=1M if=/dev/sd?? of=IMAGEPATH.img

When the copy is complete, you can shut down the Pi, or safely remove the hard drive card from your main computer, plug it back into the Pi or swap your SD cards back to normal, make sure everything’s hooked back up like it was before, and restart the system.

Restoring from a backup image

If you need to restore your system to a previous state, the process is just the same, but with the “if” and “of” parameters flipped like they were when you created the original SD card. You’ll need to read from the image file and write it to the hard drive partition.

Shut down the system and move the hard drive to the computer that will be doing the restore, or swap to the “backup” SD card if you’re doing this directly from the Pi, and run the dd or dcfldd command as appropriate. Don’t forget to mount the data partition if you’re doing this from the same Pi. You might want to edit the /etc/fstab file so that this happens automatically whenever you’re using your “backup” SD card.

Note: This post is part of a series. Each post builds on the previous ones. If you are just trying to add one thing to an existing system that was not built following this series, then I cannot promise that these instructions will work for you, although they probably will. If you’ve started from something other than a non-NOOBS Raspbian image, then you’ll probably need to adjust for that.

Please refer to the series Introduction for a list of all the different posts in the series.

Self-Promotion: I have recorded this series as a screencast for Pluralsight:
(http://www.pluralsight.com/courses/raspberry-pi-home-server)
If you have a Pluralsight subscription, please consider watching it. Reading the instructions is one thing, but watching it done demystifies the whole process.

Thank you!

Running the Raspberry Pi off of an SD card is simple, affordable, and very convenient. SD cards are also very small compared to hard drives, though, and they can only be written to a finite number of times. That number is ridiculously large, but it can still be used up faster than you think by things like virtual memory swap files which are written and re-written constantly.

If you plan to run this server 24/7 and actually use it, you’re probably going to want something a little more robust. My current setup is using a 120GB SSD for the system drive. I’m using an SSD because it can be powered entirely from the Pi’s USB port. I could have gone with something smaller, both storage-wise and physically by buying a small MSata drive and an adapter, but in the end it was cheaper to buy a normal 2.5″ SSD and put it in a case I already had.

This SSD has two partitions on it. The first is the “os” partition, which I left free back in the post “Adding a Hard Drive“. The second is my “data” partition, and holds information about my collection such as MiniDLNA’s database. I don’t actually put my media files or public share on the data partition anymore. My public share now lives entirely on an external RAID enclosure. In the original version of this post, I used the RAID box as the system drive, but over time I have discovered that RAIDs don’t work very well in this capacity, so I just use the RAID for my shares now. Everything that is the Raspberry Pi Home Server itself lives on the SD card and on the SSD.

Your setup doesn’t need to have as many moving parts, though. A typical Raspberry Pi Home Server setup might consist of a Raspberry Pi attached to a single, very large, hard drive. You can get 5TB external drives for under $150 these days. You only need to partition off a small space for the Pi to boot from, and the rest is yours to store most, if not all, of your stuff. This is how my CrashPi works.

Booting from a hard drive

Okay, I’ll level with you. You can’t actually boot entirely from the hard drive, at least not yet. The Raspberry Pi 3 has a USB boot mode, but it’s still considered experimental at this time and you have to do some tweaks to even make it available. You can boot mostly from the hard drive today though, and it should be easy enough to adapt this post in the future when the Pi 3’s alternate boot modes are considered finished.

When the Raspberry Pi boots up, it looks to the first partition on the SD card for instructions on what to do next. That boot partition contains just a tiny piece of the boot process. You can easily move all the stuff that comes after that first step to a different device, such as the external hard drive. All that stays behind on the SD card is a very small boot partition, everything else moves to the hard drive.

The resulting system should run faster and smoother, and you won’t be using up your SD card’s limited number of write cycles on high-frequency stuff like a virtual memory swap file.

I’ve added my own bits, such as consulting the boot/cmdline.txt file rather than simply assuming the root file system’s location, using UUIDs to identify the partitions, and expanding the existing swap file rather than creating a separate swap partition.

I’ve turned these resources into a walkthrough that matches the rest of this series, but I wanted to at least point out where my source material originated.

Shut down services

We’d like as little to be going on with this partition as possible while we’re copying it. If you have another Linux system, or another Raspberry Pi handy and a USB adapter for your SD card, then your best shot is to let another system do this copy, but for this tutorial, I’m assuming you only have the one Pi available. To make sure things are as “quiet” as possible, we’re going to want to exit the desktop, shut down any of the background services we’ve installed, and do the copy from the command line.

First, let’s boot to the command line. If you’ve configured the Pi to boot to the desktop, then unfortunately there’s no option to simply exit the desktop, so we’ll need to tweak the configuration first. Use the desktop configuration tool or raspi-config to set the pi to boot to the command line.

Reboot the Pi, and connect via SSH using a tool like PuTTY, or connect the Pi to a monitor and work directly from the Pi. I’ll be using PuTTY for this demo.

Once you’re logged in, shut down any of the services you’ve installed so far. For me, that’s Samba, MiniDLNA, OpenVPN, Resilio Sync, and NUT. If you have any other services like CrashPlan or Transmission running, you’ll want to stop those too.

We want as little activity going on as possible while we clone the OS partition.

Copy the old root partition to the new drive

The file /boot/cmdline.txt specifies the location of the root filesystem. You’ll need to edit this file later in order to boot from the hard drive, but for now, you just need to know for sure where the root filesystem currently resides. Show the cmdline file’s contents like this:

cat /boot/cmdline.txt

The part immediately to the right of “root=” specifies the device and partition that holds the root filesystem, and that’s the partition whose contents you need to move to the hard drive. In my case, it says “mmcblk0p2”. If you started from a raw Raspbian image, then yours probably says the same thing. I can’t guarantee that the default Raspbian image might not change in the future though, so it’s best to check.

Copy the existing root partition’s contents to the new 16GB partition (sda1) on the hard drive, changing the highlighted part to match the current root filesystem path from above.

Important: Be very careful here. Make sure that sda1 is the partition you set aside for the OS. If you configured your server differently, then double-check before continuing.

sudo dd if=/dev/mmcblk0p2 of=/dev/sda1 bs=32M conv=noerror,sync

This can run for quite a while, depending on how large your SD card is. Unfortunately, the dd command doesn’t give you any kind of feedback. In this case, no news is good news. As long as the drive appears to be busy, then keep waiting.

Here’s what that all means:

“dd” copies things

“if” specifies the input file, in this case an entire partitions

“of” specifies the output file

“bs” specifies how many bytes to copy in one chunk, here it’s 32 MB

“conv” specifies how to convert things as they are copied

noerror says to continue if anything goes wrong

sync does some padding during the copy

When the copy process eventually finishes, check the target partition on the hard drive for errors.

sudo e2fsck -f /dev/sda1

Press enter at the prompts to fix any errors that it finds. You can probably expect one about the free block count, and one about the free inodes count.

Since the image that you just copied over from the SD card needs to be smaller than the partition you copied it to, you’ll want to expand it to fill up the available space.

sudo resize2fs /dev/sda1

Uniquely identifying a drive

This next section is a partial repeat from the “Adding a Hard Drive” post, but I’ll include it again for completeness, and so that you don’t have to bounce back and forth between the posts.

Warning: Because the next section involves copying around large strings of unmemorizable data, I recommend performing the steps through a remote SSH window, or from a terminal on the X desktop. The ids you’ll need to copy must be copied verbatim, letter for letter, or you’ll find yourself unable to boot, and you’ll have to start over again from your last backup.

Currently, the cmdline.txt file specifies that the root file system is on the second partition of the internal SD card (/dev/mmcblk0p2). We’d like that to say “/dev/sda1″ instead, but there’s a problem. As I mentioned earlier, we don’t have any guarantee that this particular drive will be called “sda” in the future. What we need is a way to uniquely refer to this drive no matter what letter it get assigned on any given day.

This was why we built the drive with a GUID Partition Table. Each partition on a GPT device is assigned a universally-unique identifier (UUID).

Note: The difference between GUID and UUID is not important here, they are different terms for the same thing; a very long, randomly-assigned number.

Get the partition UUID for /dev/sda1

sudo blkid /dev/sda1

The piece of information we’re interested is the “PARTUUID” value. Take a picture, write it down very carefully, or select and copy the text if you are doing this through a remote terminal connection like I am.

Boot Configuration

The new root filesystem’s partition is now uniquely identifiable, but the Raspberry Pi doesn’t know to use it. Edit the /boot/cmdline file to change where the bootloader will look for the root filesystem.

sudo nano /boot/cmdline.txt

Find the part that says “root=/dev/mmcblk0p…” and change it to “root=PARTUUID=” and whatever your Partition unique GUID was from above. Also add the string “rootdelay=5” at the end. This will give the Raspberry Pi time to discover the USB drive before it tries booting from it.

The result should look like this (sorry for the small picture):

Exit Nano, saving your changes (ctrl-x,y,enter)

Reboot (Important, do not skip this step)

Before the next set of changes will “stick”, you’ll need to reboot so that the Raspberry Pi uses the hard drive for the initial load. If you don’t reboot now, nothing you’re about to do will count, and you’ll just have to do it all again. You’ve been warned.

sudo reboot

If everything goes well, you should find yourself back at a login, and you can continue. If something went wrong, go back to your most recent backup and try again.

Mounting filesystems

Partitions and filesystems are not the same thing. Linux now knows what partition to load the OS from, and that’s great, but as things stand right now, it’s still going to mount the root filesystem from the SD card.

To complete the transition to the hard drive, you’ll need to edit the filesystem table. This file controls what gets mounted where, and in what order, and it needs to know where the root filesystem is. Take a look at the current contents.

cat /etc/fstab

Note: You’ll probably have an entry here for your data partition that we added earlier in the series.

That third line is the root filesystem, which you can tell by the “/” in the second column. Unfortunately, it’s still loading from the 2nd partition on the SD card (mmcblk0p2). You could change this to say “/dev/sda1”, but that would only work as long as the drive continues to get the name “sda”.

Fortunately, you can use a very similar UUID-based trick here to uniquely identify the filesystem no matter what letter the device gets. Filesystems have UUIDs too, and you can see them all with this command:

sudo blkid

If you look carefully, you may notice a problem. both /dev/mmblk0p2 and /dev/sda1 have the same UUID. So much for being unique, right? This is because of the way we cloned the old root filesystem into a new location. It brought the whole filesystem over, including its UUID. Before you can use a unique Id to identify the drive, you’ll need to make sure it’s actually unique.

You need to give /dev/sda1 a new UUID. You can hand-assign your favorite UUID, or just let the computer pick a random one. Use the following command to assign a new UUID to the first partition on the hard drive:

sudo tune2fs /dev/sda1 -U random

Note: At the time of this edit, there seems to be an issue with the newer “Jessie” release of Raspbian, and it may prevent you from changing the UUID on the partition. If you see an error that says “The UUID may only be changed when the filesystem is unmounted.”, even though it’s NOT mounted, then you’ll need to perform one more tweak. The magic words are

sudo tune2fs -O ^uninit_bg /dev/sda1

I can’t take credit for this one at all. I just found it here. After reading the man page on tune2fs, I still don’t honestly even understand what this has to do with anything, but it does seem to do the trick, so if you get the above error, give this command a shot, then try the “-U random” command again and it should succeed.

Either way, display the device Ids again, just to be sure

sudo blkid

Copy down the UUID value (not the PARTUUID value) for /dev/sda1. You’ll need it in a minute. Open the filesystem table in an editor.

sudo nano /etc/fstab

Change “/dev/mmcblk0p2” on the third line to “/dev/disk/by-uuid/” and the new UUID you just assigned to /dev/sda1.

The end result should look like this:

Your ids will obviously differ, but the important thing is that you have pointed the root filesystem to the hard drive. Any other mounts you’ve defined should appear here as well. Don’t worry about the columns lining up, it doesn’t matter, but I’m presenting mine to the public, so I’ve gone ahead and made it pretty

Exit Nano, saving your changes (ctrl-x,y,enter)

That’s it. you’re ready to reboot again, and this time, everything should be faster.

sudo reboot

You’ll notice that the activity light on the Raspberry Pi will not blink much anymore. That’s because the SD card is no longer being accessed for anything other than the boot partition. The hard drive’s activity light will now blink where the Raspberry Pi’s activity light used to.

When the system has rebooted, double-check your filesystem table to make sure your changes are still there.

cat /etc/fstab

If you see /dev/mmcblk0p2, and no line for your data partition, it’s because you skipped that “Reboot” step above. See? I told you it was important. Redo everything in the “Mounting filesystems” section, and reboot again.

Swap configuration

One thing you may have noticed if you’ve looked at other walkthroughs for booting from the hard drive is that they usually create a “swap” partition on the hard drive to be used as virtual memory.

If you look back at the filesystem table from the beginning of this post, though, you’ll notice that Raspbian never had a swap partition in the first place. That’s because Raspbian is set up to use swap files instead of swap partitions. Raspbian’s swap file lives at /var/swap, and since we just moved the whole root filesystem onto the hard drive, the swap file came along for the ride.

At this point, you’re already running your swap file from the hard drive, and you didn’t even have to do anything. Check it out with the “swapon” command:

sudo swapon –s

This shows the swap summary, which will tell you what swaps are in use. It should have one entry in it (/var/swap). It’s pretty small, though; only 100MB:

The partition we created for the root filesystem is 16GB (or more if you so chose). There’s plenty of space left to expand the swap file to something roomier.

Edit the swap file configuration:

sudo nano /etc/dphys-swapfile

Find the line that says “CONF_SWAPSIZE=100”. This is the size of the swap file in megabytes. Change the value to 2048, which will create a 2GB swap file, which is a more appropriate for a server running on a Raspberry Pi 2 or 3.

Close Nano, saving your work (ctrl-x,y,enter).

We’ll need to reboot one more time to get this to stick, but there’s another change to make first. If you’d like to return to booting to the desktop, now is the ideal time to set that back up using raspi-config.

sudo raspi-config

Go to “Boot Options”, then “Desktop/CLI”, and set it back to “Desktop”. When you exit raspi-config, it will offer to reboot. Go ahead and let it.

Once the system has rebooted, check the swap file by typing “swapon –s” again. You should see a table similar to the first time you ran this command, only now the swap file is much larger:

Wrapping up

You now have a Raspberry Pi that boots (mostly) from an external USB drive. It also uses this drive for its virtual memory swap file. The whole system should run more smoothly now, and you won’t have to worry about using up your SD card, if that’s the sort of thing you worry about.

Although most of the important stuff is on the hard drive, you should probably make at least one more backup of the SD card for safety.

What’s Next?

In the next post, I’ll show you how to make backups now that the SD card isn’t being used for the root filesystem anymore.

Note: This post is part of a series. Each post builds on the previous ones. If you are just trying to add one thing to an existing system that was not built following this series, then I cannot promise that these instructions will work for you, although they probably will. If you’ve started from something other than a non-NOOBS Raspbian image, then you’ll probably need to adjust for that.

Please refer to the series Introduction for a list of all the different posts in the series.

Self-Promotion: I have recorded this series as a screencast for Pluralsight:
(http://www.pluralsight.com/courses/raspberry-pi-home-server)
If you have a Pluralsight subscription, please consider watching it. Reading the instructions is one thing, but watching it done demystifies the whole process.

Thank you!

My previous home “server” before embarking on this series was just an old laptop of mine. It wasn’t particularly strong, but it could run CrashPlan and serve files. One advantage it had was that, being a laptop, it had a built-in battery and knew how to shut itself down if the power went out. Of course, I had to go downstairs and start it back up once the power was restored, but at least nothing got corrupted.

If you have a desktop computer at home or at work, you may also have an Uninterruptable Power Supply (UPS) under your desk. This is basically a battery big enough to power your computer and monitor long enough for you to save what you are doing and shut down gracefully. Most modern UPS units also have a USB port on them and come with software so that when the power goes out, the UPS can tell the computer to automatically suspend, hibernate, or shut itself down depending on how you’ve configured it.

It shouldn’t surprise you that there are UPS options available for the Raspberry Pi as well. Take the CW2 “Pi UPS” (http://www.piups.net), for example. This product will work fine for most simple Raspberry Pi projects running off of an SD card, but my particular installation has a RAID enclosure hooked up to it. I don’t think AA batteries are going to cut it. What I need is something that will keep the hard drives spinning long enough for the Pi to shut down safely.

I’ve had my Raspberry Pi Home Server plugged into a UPS for well over a year now, and it survives power outages just fine, shutting itself down safely, and restarting automatically. My Raspberry Pi Home Server is attached to a CyberPower SX550G. I chose this model because:

It has a USB connection to the computer.

It was on sale for $40.

I’m cheap.

It’s a 550VA battery backup, which probably wouldn’t get you very far with a full-sized desktop computer, but it should be more than adequate for a Router, a Raspberry Pi, and a couple of drives. It could probably run those devices for a good couple of hours if needed, actually. I haven’t done the math, but if not for the external drives, it could probably run the Pi for days.

So how can we get the Pi to use it intelligently? It certainly didn’t come with a ready-made software package for the Pi, although Linux software is available for it.

Network UPS Tools (NUT)

The Network UPS Tools package, aka “NUT” (http://www.networkupstools.org), is a collection of programs meant to make UPS hardware from different manufacturers work in roughly the same way. It’s available for a wide variety of platforms, one of which happens to be Debian Linux, of which Raspbian is a variant.

NUT consists of three major components

A driver to communicate with your particular UPS using whatever protocol it supports, and translate that into a common API.

A daemon (service) that connects via the driver and acts as a communication hub.

A client program that can perform various tasks such as shutting down the computer when the power state reported by the daemon own sitechanges.

There are a few other components, such as command-line utilities that can tell you about the current state of the UPS, or allow you to make changes to the UPS configuration. For the most part though, we’re only concerned with the three components I just mentioned.

Note: NUT works with a wide range of UPSes, but without owning one from each different manufacturer and/or vintage, I can’t create instructions specific to each model. Questions about how to get NUT working with your particular UPS are best addressed on the project’s own site, or its GitHub site. For the most part, though, they should all work the same way.

If your UPS is older, and doesn’t have a USB port, but has a 9-pin serial port, you are totally on your own. I tried to make an older serial-ported UPS work through an RS232/USB adapter, but as it turns out, the UPS didn’t actually communicate over the serial port. It just shorted two pins together to mean that everything was okay. Maybe you could hack something together using the GPIO pins. If so, you’re a better man than me.

Installing Network UPS Tools (NUT)

Simplicity itself. Like most things we’ve installed in this series, NUT is available through apt-get

sudo apt-get install nut

Select your driver

Find the driver for your particular UPS by looking at the compatibility list on the NUT site (http://www.networkupstools.org/stable-hcl.html). If you can purchase a UPS that’s on the list, then that’s great. If you don’t find your UPS on the list, that doesn’t mean it won’t work, though. For instance, my UPS isn’t on the list, but a ton of other devices from the same company are, and they all seem to use the same driver (usbhid-ups). I gave it a shot, and it works just fine. YMMV

Configure the UPS

Once you’ve identified the driver for your UPS, you’ll need to edit the ups configuration file.

sudo nano /etc/nut/ups.conf

This file, like all of the NUT configuration files, is very well documented inline, and explains all of its different options. At the bottom of the file, you’ll add an entry for your UPS. You’ll need to give it a name, specify a driver, and give it a description. There is also an option to specify a port. For units that connect via a USB cable, this is meaningless, but it’s still required, so use the value “auto”.

[RPHS]
driver = usbhid-ups
port = auto
desc = "CyberPower SX550G"

The name in square brackets is so you can tell multiple UPS devices apart in case you have one NUT server monitoring multiple devices. Unless you’re building something really esoteric, you probably only have one UPS like me, so for lack of anything better to call it, I’ve named mine “RPHS” after the computer it serves. You can put anything you want in the description, so I just put the make and model of the device for reference. The end result should look something like this:

That’s it for configuring the UPS itself. Close and save the file (ctrl-x, y, enter).

Configure the daemon

A daemon is the Unix/Linux term for any invisible background process. Windows folks call these “services”. For NUT, there is a daemon that is in charge of listening to the UPS via the driver, and telling the client applications what to do. Edit its configuration file like this:

sudo nano /etc/nut/nut.conf

For this simple application, where both the client and the server programs will be on the same computer (the Pi), go to the bottom of the file and set the MODE to “standalone”.

Close and save the file.

Verify hardware configuration

To check whether the driver and daemon are configured correctly, you can simply start up the service.

sudo upsdrvctl start

The first time I tried to connect to the UPS I got the error “could not detach kernel driver from interface 0: Operation not permitted”. I rebooted and tried again, and everything was fine.

Notice that this time I got a message about “Duplicate driver instance detected”. This is because now that I have things configured correctly, the driver started up automatically when I rebooted. Not only that, but the daemon should be running now as well. Check it like this:

sudo service nut-server status

You should get a message that the NUT server is running. You can now ask the NUT server questions about the status of the UPS using “upsc”, one of several command line utilities that apt-get installed. Substitute the name you gave your UPS above as needed.

upsc rphs

You’ll get quite a long list of information about the configuration and status of your UPS.

Depending on the make and model, you’ll get more or less detailed information.

Configure the monitor

That’s two out of the three layers. Last but certainly not least is the “upsmon” client. This is the part that will actually shut down the computer when the NUT server says so.

Define credentials that the monitor client program will use to connect to the server.

sudo nano /etc/nut/upsd.users

Go to the bottom and define two users, one called “admin” and one called “upsmon”. You can actually name them anything you want, but naming the “monitor” user after the program that will use it (upsmon), seems to be the convention. You give the “admin” user rights to issue commands and change configurations with the “actions” and “instcmds” settings. The “upsmon” user has no such rights, it’s just there to listen and shut the computer down when the power goes out.

Since this is the computer that’s actually in charge of monitoring the UPS, set the “upsmon” setting to “master”.

Next, edit the configuration file for the upsmon client program.

sudo nano /etc/nut/upsmon.conf

This configuration file is pretty long, and has a lot of options. The one we’re interested in is about five pages down. Look for the example lines that start with “MONITOR”, and create a new entry on the blank line below that section. There are six parts to this setting.

The keyword “MONITOR”. It does have to be all uppercase, by the way.

The “system” name in the format UpsName@HostName. I called my UPS “RPHS”, and since we’re running in standalone mode, we can just use “localhost” for the host name, so the resulting “system name” is “RPHS@localhost”.

The “power value”. This only applies to big servers with multiple redundant power supplies. Just set it to “1”.

The user name that you established in the upsd.users file (upsmon) .

The password that you established in the upsd.users file (mypasswd).

A value indicating whether this computer is the master or slave. You can read about the distinction in the upsmon.conf file itself, but for a standalone system like this, use “master”.

When you’re done, it should look something like this.

This next part is optional, but handy for testing that your system is working. Scroll down a bit further, and look for the NOTIFYFLAG section. There are two entries there, “ONLINE” and “ONBATT”, but they seem to be commented out by default in the current installation. Remove the pound sign from the beginning of these two lines, so that NUT can tell us about the state of the UPS when it changes.

Close and save the file.

Next up, a little permissions wrangling. You need to set up the various configuration files to be readable by the NUT components that use them, but not by other users. This prevents anyone from reading the password, and sending unauthorized commands to the server to shut everything down. It may be overkill for a simple home network, but it’s also really simple to do.

UPS Commands

Depending on the sophistication of your particular UPS, you may be able to send it commands to do things like initiate a self-test, or simulate a power failure. You can get a list of what your particular UPS supports with

sudo upscmd -l rphs

The number and type of commands will vary by manufacturer and model, so your output may not match mine. There are a few commands here worth mentioning, though. The “load.off” command will shut down the UPS immediately. Think of it as the “goodbye world” command. You generally don’t want to mess with that one. You could actually use this command to turn a second UPS on and off, allowing your Pi to control the power to something else. I’ll leave that one up to your imagination, but there are certainly cheaper ways to automate things.

The “beeper.mute” command will temporarily silence the warning beep that my UPS makes when the power goes out. This will make testing the system a bit less annoying here in the next section. The “beeper.disable” command would probably have the same effect, but on a more permanent basis.

Testing the system

Restart the NUT service and client daemons to ensure that they load all of the configuration changes we’ve made.

sudo service nut-server restart
sudo service nut-client restart

At this point, you can unplug your UPS from the wall and, after a short delay, you should get a message that the system is now running on battery power.

IMPORTANT: Wall messages like this don’t seem to show up in Terminal windows on the desktop. If you want to test this part for yourself, connect via SSH using a program like PuTTY on Windows.

If you plug it back in, you’ll get another message that power has been restored.

So far, so good. Now for the real test. Unplug the UPS from the wall and leave it unplugged. If your UPS supports it, now is a good time to issue that “beeper.mute” command. In my case, I issued the “upscmd rphs beeper.mute” command, and was prompted for a user name and password. We established the user name “admin” in the config file above.

You can sit and stare at the screen, wondering when it will shut down, or you can periodically check on the status to see how fast you’re using up the battery. I plugged my laptop charger and a television into the UPS to speed things along. A good, old-fashioned 100-Watt bulb would help, too.

Eventually, when the battery gets low enough, the system should shut itself down.

Plug the UPS back in, and if you’re lucky, everything will start back up again. For some models, even after the power is restored, you may have to physically press a button on the UPS to start everything back up. Mine starts up on its own, so a few minutes later I was back up and running.

Email notifications (optional)

Wall messages are just fine if you’re connected via SSH or looking at a monitor connected directly to the Pi, but if your server is headless or you’re running the desktop, you won’t see them. We need a way to be notified when something goes wrong. NUT can be configured to call any arbitrary command when it has something to say, so we’re going to set it up to send out email messages. This can inform you of power issues even when you’re not connected to the Pi at all.

We’re going to need to install a couple utilities to make this happen. I’ll be using ssmtp because it’s the simplest thing that works. Because it only knows how to send emails, it doesn’t leave any services running in the background using up CPU time. If you’ve already configured some other kind of mail agent on your system, then you’ll need to adjust accordingly.

sudo apt-get install ssmtp mailutils

Next, we’ll edit ssmtp’s configuration:

sudo nano /etc/ssmtp/ssmtp.conf

For this demonstration, I’ll be configuring ssmtp to work with gmail. If you want to use your ISP’s email services, then check out the ssmtp documentation, but most of the concepts should be the same.

Change the “mailhub” setting to point to your email server. For gmail, that will be “smtp.gmail.com:587”. For gmail, you’ll also need to add the “UseSTARTTLS=YES” setting. Scroll to the bottom of the file and add settings for “AuthUser”, “AuthPass”, and uncomment the “FromLineOverride”. The result should look something like this:

Note: If you have two-factor authentication set up with gmail, then the AuthPass setting will need to be an application-specific password. You can read more about that here.

Test your email setup by sending yourself an email from the command line.

echo "Test body" | mail -s "Test subject" [YOUREMAILADDRESS]

Check your inbox for the email. If you get it, then you’re ready to configure NUT to use ssmtp. Some wireless providers have special email addresses you can use to send texts to your phone. I have configured my NUT installation to send me texts rather than plain old emails, so I’ll notice them immediately.

Next, edit the NUT client configuration one more time to set up the email notifications.

sudo nano /etc/nut/upsmon.conf

Scroll down to find the NOTIFYCMD section, and add a line that says

NOTIFYCMD /etc/nut/upssched-cmd.sh

You presumably don’t want to be notified about every single power event that happens, so scroll down to the NOTIFYFLAGS section again, and add “+EXEC” to the end of the messages you want to receive emails for. I’m going to enable emails for the ONLINE and ONBATT events. The end result looks like this:

All that’s left is to create the script that we named in the NOTIFYCMD setting, “upssched-cmd.sh”.

sudo nano /etc/nut/upssched-cmd.sh

Paste in the following, substituting the email address where you want the notifications to go.

Test the new script by calling it in the same way that NUT will in the event of a power event.

sudo /etc/nut/upssched-cmd.sh "This is a test message"

If you got that message, then everything is configured. Restart the NUT client so that it will pick up on the configuration changes.

sudo service nut-client restart

And now you can test the system again by unplugging the UPS from the wall. If everything is configured correctly, you should get an email (or text message) telling you that the power has gone out. Plug the UPS back in, and you should get another message telling you about it. Now you’ll know about power problems even when you’re not at home.

Adding more Pis to the same UPS (optional)

With Raspberry Pis being so cheap, you may very well have more than one. Perhaps you’ve split CrashPlan or MiniDLNA off onto a Pi all by itself. Maybe you went a little crazy and built a Pi-powered super-computer cluster. If, for any of these reasons, you have multiple Pis, and you’d like them to share the same UPS, how will they all know when the power’s gone out? After all, you can only plug the USB cable into one of them, and buying a separate UPS for each Pi would be overkill, right?

The great thing about NUT being split up into layers is that you can run multiple client instances against one server. Normally, this is the sort of thing you’d only see in a large server rack, where you’ve assigned one computer to watch the battery and tell all the others when the power goes out. It’s not the sort of thing regular home users usually care about, but Raspberry Pi owners are not regular home users.

If you do happen to have multiple Pis, getting them all to share a single UPS and shut down gracefully is actually really simple.

Reconfigure NUT on the server

Earlier, we configured NUT in “standalone” mode, which means that the server and client were on the same machine. It also means that the server isn’t expecting any other clients to be talking to it, so it simply isn’t listening for them. With a little bit of reconfiguration, we can tell the NUT server to listen for clients other than itself.

Edit the NUT configuration file.

sudo nano /etc/nut/nut.conf

Scroll down to the bottom, and change the mode from “standalone” to “netserver” (not “nutsever”, mind you).

Close and save the file (ctrl-x, y, enter)

Next, you’ll need to configure credentials for the remote machines to use when connecting to the server.

sudo nano /etc/nut/upsd.users

Add a new user at the bottom. You can call it anything you like, but I’ve chosen “upsmon-remote”. Give it a password, and specify that it is a upsmon slave.

Next, you need to tell the NUT server to listen for client connections. Edit the UPS daemon configuration file.

sudo nano /etc/nut/upsd.conf

Find the “listen” section, and add a listen directive using the IP address of your server. If you don’t configure anything, the NUT server will listen only to the computer that it’s on, which is localhost, or IP address 127.0.0.1. Since you now want NUT to listen for clients on other systems as well, you’ll need to specify a real IP address.

Adding the server’s own IP address might seem a little redundant, but remember that NUT is made to work on all kinds of equipment, and it’s very common for larger-scale servers to have multiple network interface cards, and you’d only want it to listen for NUT traffic on the “inward” facing one.

Because you will still want to run a ups monitoring client locally on the server itself, and you’ve now overridden the defaults by specifying one IP address, you’ll need to explicitly add an entry for localhost (127.0.0.1) back in as well, otherwise the local ups monitor will stop working. The result should look something like this:

Note: You’ll need to substitute your own Pi’s address in place of 192.168.1.11. That’s just the address where my own Pi server lives. The default port that NUT listens to is 3493. If you wanted to change that, for some reason, you would do that here. I’m going to leave it at the default port, though.

Reboot the server to get things reconfigured and restarted.

sudo reboot

Double check that you can still communicate with the NUT locally:

sudo upsc rphs

You should get the same kind of listing of UPS settings and status as you did in standalone mode:

If you get some kind of error instead, make sure that you added an entry for localhost (127.0.0.1) in addition to the Pi’s regular IP address.

You can actually run NUT configured this way, but without any clients connected, and it will behave in pretty much the same way as it did in standalone mode, but it will still be listening for additional clients over the network.

Configure the client machines

Install NUT on each additional Pi that you intend to run from the UPS.

sudo apt-get install nut

For these additional client-only installations, you can skip configuring a UPS driver and proceed directly to configuring the client.

sudo nano /etc/nut/nut.conf

Go to the bottom of the file, and set MODE to “netclient”.

Next, edit the upsmon configuration file.

sudo nano /etc/nut/upsmon.conf

This time, configure the client to talk to the remote server for information, and configure it as a slave. This time, the “system name” will not be “RPHS@localhost”, but “RPHS@192.168.1.11” (again, substitute your server’s IP address).

If you remember from earlier, the first part of this “system name” is the name you gave your UPS, and the second part is the address of the server it’s attached to. The name and password should match the “upsmon-remote” name and password from above, and the final parameter will be “slave” instead of “master”. The result should look like this:

Note: My other Pis are all busy right now, so I’m doing the edits (but not saving them) on my original server for these screenshots. Please ignore the name in the title bar.

The difference between “master” and “slave” has to do with when each one will shut down. When the UPS says the power has dipped below the safe threshold (20% for my UPS), the server (the master), will tell all of the clients (the slaves) to shut down first before shutting down itself.

Like on the server, you’ll need to set up the permissions so that the NUT configuration files are readable by the nut user. Unlike the server, however, you don’t need to touch the upsd.users file because users are defined on the server, not the client.

sudo chown nut:nut /etc/nut/*
sudo chmod 640 /etc/nut/upsmon.conf

Check connectivity

From the slave system, you can issue the same kinds of commands you could run from the master system. Use this to verify that everything is communicating properly.

sudo upsc RPHS@192.168.1.11

I’ve used the IP address because name resolution didn’t work for me. It’s one of those things I’ll probably learn to set up someday, just not today.

Test the system

Once again, unplug the UPS, and you should see warning messages appear on both the server and client(s) right away. Again, you may want to mute the beeper if your hardware supports it, and plug something in to help the battery drain faster.

Apart from the name of the server sending the message, and the fact that the client shut down first, you wouldn’t be able to tell the difference between the client and server installations, even though only the server is actually attached to the UPS.

Lather, Rinse, Repeat

Repeat as necessary for each additional Raspberry Pi you are plugging in to the same UPS. That’s all there is to it. When the power goes out, all your Pis can now shut themselves down safely.

Wrapping up

You know the drill by now. Shut down the server, and back up the SD card.

What’s Next?

In the next post, we’ll reconfigure the Pi to boot from the hard drive instead of the SD card.