Post navigation

Problem: You have a computer running Debian 7 wheezy (also Debian 8 jessie) with an encrypted root filesystem. At startup you’d like to be able to unlock the disk over ssh, maybe because it runs headless.

There are some existing solutions to this: early-ssh worked on previous Debian versions. There are other blog posts showing some methods but I found them unwieldy (ie a 35 second delay before you can type your passphrase.)

Here’s how I’m doing it. Unless otherwise mentioned, all of the following commands need to be run as root. The server name in the examples is “debian_headless”

At time of writing I was using versions Debian 7.0 Wheezy with packages cryptsetup 2:1.4.3-4, initramfs-tools 0.109.1 and dropbear 2012.55-1.3.

[EDIT Jan 2015: Page now updated with a small tweak that makes it work perfectly on Debian 8/Jessie also. Thanks to olze in the comments for figuring that out.]

These steps may work on recent Ubuntu versions as well, please let me know if you try them out. [EDIT: after some discussion in the comments, we determined Ubuntu’s busybox packaging means that it won’t work without more tweaks.]

Overview

The very first thing a Debian system runs at startup is a minimal initramfs ramdisk which performs basic initialisation, mounts the main filesystem and starts the rest of the boot process.

This is the stage when an encrypted rootfs gets mounted. We need to customise it so we can ssh in and unlock the rootfs.

Install prerequisites

apt-get install dropbear initramfs-tools busybox

Dropbear is a slimline SSH server. This package installs into the initramfs and runs during early boot only, allowing you to log in remotely.

Authorise the user(s) you want to be able to login as root and unlock

Hopefully you’re already using SSH Key Authentication, otherwise have a read about it and why it’s a good idea. Set it up for the user(s) on the client systems by having them run ssh-keygen.

You have to add the public keys for these users to Dropbear’s list of authorized_keys for login.

For each client user you want to allow to unlock, take their public key file ~/.ssh/id_rsa.pub and append it to /etc/initramfs-tools/root/.ssh/authorized_keys

Alternative: Have a unique key for root/unlock

If you don’t want to use your usual user-level ssh key and passphrase to provide potential root-level access to your headless machine, generate a separate key (with a different passphrase) and then copy it across to authorized_keys instead:

See the nfsroot kernel docs for full details. Remember if using DHCP that you’ll need a static DHCP entry configured so you can predict what IP your headless machine is going to have on startup.

Annoyance: Interface doesn’t reconfigure

It seems that after the main system boots, the network interface won’t automatically change to whatever is set in /etc/network/interfaces (because the interface is seen to be already “up” it gets left alone.) To work around this you could put a line ifdown eth0; ifup eth0 in /etc/rc.local, or just keep the initramfs network settings the same as the ones in /etc/network/interfaces.

EDIT: bikepunk posted in the comments with a correct workaround for this. Add the following line to /etc/network/interfaces under your primary interface’s heading:

pre-up ip addr flush dev eth0

Unlock Script

I’ve written a little initramfs-tools hook that generates scripts for easy unlocking of the rootfs.

Download this script file, then read it to make sure you’re not suspicious about what it’s going to do. Then copy it to /etc/initramfs-tools/hooks/mount_cryptroot and mark it executable:

chmod +x /etc/initramfs-tools/hooks/mount_cryptroot

There’s a section at the top of mount_cryptroot that allows you to set an ALLOW_SHELL variable if you want to be able to drop to a remote root shell for diagnostics. Note that this explicitly allows a remote root shell to anyone with an authorised SSH private key. I can’t guarantee it’s impossible to get to a root shell otherwise, but it’s at least not explicitly welcomed.

Update the initramfs

Update the initramfs so it includes all your changes:

update-initramfs -u

Add client SSH config entry

Because Dropbear has a different SSH host key to the normal system, OpenSSH will (rightly) complain if you try connecting to it on the same IP. The solution is to create a special host definition for the initramfs environment. A custom section in ~/.ssh/config:

Now you can log in to the unlock stage by simply typing ssh headless_unlock. (Replace headless with whatever name you want to use.)

[EDIT: Benjamin has pointed out in the comments that it’s possible to have the same host key in Dropbear and OpenSSH together. So you can follow his steps instead, if you’d like the same host key for both cases. Thanks Benjamin!]

If you authorized a unique private key to use for unlocking, you can add another line IdentityFile ~/.ssh/id_rsa_headless or similar.

Putting it all together

Here’s what it looks like to connect to my headless host and unlock the disk:

Final word about security

When dealing with remote access and with encryption, it’s always worth thinking about security tradeoffs.

There’s the obvious risk that enabling Dropbear in the initramfs means allowing remote root logins to your system. Even with ALLOW_SHELL turned off there’s no guarantee that someone with access to an authorized SSH key (or Dropbear exploit) can’t log in, pop a root shell somehow, and cause havoc.

In addition, when I was looking for existing techniques to do this I came across a post exclaiming ssh unlocking was a terrible idea, because someone with physical access to your machine can then impersonate you remotely and you’ll never know.

I think this is worth considering, but it’s not as bad as it might seem. All encrypted disk access, with local or remote unlocking, relies on physical security of your device. Yes, if someone gets physical or root access they can steal your Dropbear host key and set up another server that pretends to be you and steals your keys. However, they already have physical or root access. They can just as easily insert malware in the initramfs that quietly captures your passphrase as you type it in in locally.

In either case, you have to be sufficiently sure noone is messing with you. With remote access that’s maybe more likely, but I’m not too worried. I’m using this on a USB stick PC that fits in my pocket, and I don’t want a burglar walking out my door with all my personal data. If some very dedicated person comes after my data, there would be easier ways to get it.

(Also, an obvious disclaimer – all the advice/scripts on this page is worth what you paid for it. I’m not a security professional. Please tell me if it seems broken/unsafe, but don’t blame me if you use it for something important and it isn’t safe enough.)

Many thanks for this article, it was very helpful.
Just wondering if it’s possible to configure more than one early boot network interface with different IP address each. Say I have a server with two physical network cards: one is connected to local network, the other is connected to the Internet. The point is to be able to ssh and supply hard disk password not from local network only, but from other part of the world as well.
Can I configure two interfaces in the “export IP=…” line for this purpose?

Have you tried this on ubuntu 14.04 and busybox 1.21.1? For the life of me I can’t get this to work. I can shell in, but the busy box commands don’t work or it says, applet not found, or it just says “”Unlocking rootfs…”” and sits there.

Hmm, that’s unfortunate. I just did some poking around my Ubuntu 14.04 laptop.

On Debian, /bin/busybox is supplied to both the rootfs and the initramfs by the ‘busybox’ package.

On Ubuntu, /bin/busybox is supplied on the rootfs by ‘busybox-static’ package but on the initramfs it’s supplied by ‘busybox-initramfs’ package. It’s a totally different busybox, you can find it at /usr/lib/initramfs-tools/bin/busybox. The busybox-initramfs seems to be stripped back with no -o option (and probably missing other non-essential applets), however the busybox-static busybo has -o options! Confusing, especially when they both identify themselves as the same binary! http://pastebin.com/raw.php?i=UkxH57B9 Oh Ubuntu, you’re so silly!

There is actually a third package in Ubuntu simply named ‘busybox’, same name as Debian, but this package doesn’t replace busybox-initramfs just busybox-static. :(

It’s possible that you could simply mutilate your system by copying /bin/busybox over /usr/lib/initramfs-tools/bin/busybox, run update-initramfs -u -k all, and then everything would just work. It’s equally possible doing this will break your system and prevent it from booting!

Another option that doesn’t involve mutilating packages could be to try and override BUSYBOXDIR somewhere in /etc/initramfs-tools, because /usr/share/initramfs-tools/hooks/busybox looks to BUSYBOXDIR to find /bin/busybox…

There may also be another way to get the full arguments of the “cryptsetup luksOpen” command, which is what the “-o args” argument to ps is for, but I can’t think of one off the top of my head.

If you find a solution (elegant or not) to this problem, please let me know!

Thank you for your good guide. I have one observation:
It is possible to share the same ssh host key between the main system with openssh and the initramfs with dropbear. In fact, dropbear already does this for its configuration on the root fs during package installation. We can copy over the keys from /etc/dropbear/ to /etc/initramfs-tools/etc/dropbear/ or we can reconvert the openssh host keys:

Note that dropbear doesn’t support ecdsa which is what ssh tries first. Therefore, if a client has previously connected to the host, we must first remove the ecdsa entry from .ssh/known_hosts and connect again with:
ssh -o HostKeyAlgorithms=ssh-rsa
After a new entry with an rsa key type has been added it is no longer necessary to specify HostKeyAlgorithm.

The kernel (and initrd) is usually readable by everyone (from /boot), so any private SSH host keys you place there are therefore readable by all users of your system (unlike the originals in /etc/ssh).

The same issue applies to any secret material, such as private SSH keys for making an outbound SSH connection to an auth server to retrieve the decryption key.

@Inferno
When you get an error then:
1) you copy/pasted the file instead of downloading it (and in conjunction with some wierdo behaviour editors like vim the file is messed up)
2) your initramfs is messed up, e.g. missing drivers etc.

as linux user you should know that stuff like “asking to ask” or “blabla does not work” are absolutly not helpful.
at my jessie machine this tutorial also works, mostly.

when restarting, i get “ip-config: no response after xxx – giving up.”, but this is more a bug in syslinux. will try to use a static ip instead.

You might get some clues by enabling ALLOW_SHELL in the mount_cryptroot script, rebuild the initramfs, then use Ctrl-C to drop to a root shell? If that works then you can poke around a bit to see if the mount prompt command looks similar, and run /root/mount_cryptroot.sh manually (or with ‘sh -x’) to see exactly what goes wrong.

If you figure out some changes that make it work for Jessie, let me know and I’ll set up a git repository for the scripts so we can keep track of required changes.

got it working now, simply changing the grep luksOpen to “open” only will work. Unfortunately, this will maybe not work anymore if some application is running with “open” in its name :)
but as workaround its ok

Nice work! Looks like the cryptsetup command line arguments have gotten more sophisticated.

I expect we can probably change the line from this:

> CMD=`ps -o args | grep luksOpen | grep -v grep`

To this:

> CMD=`ps -o args | grep cryptsetup | grep -i open | grep -v grep`

… and it will work on both Wheezy and Jessie (and not likely catch any false positives). Any chance you could please try that out and let me know?

– Angus

PS The ‘plymouth’ parts are for when graphical boot environments are enabled, the while loop will either match that plymouth command or the direct cryptsetup one (which is for when a text boot environment is enabled).

It took 4 months (eck), but I finally tested the change on Wheezy and it worked fine. The mount-cryptroot script download is updated so downloaders will now get a version that should work on both Wheezy and Jessie.

Hi Angus,
I am using the same versions for dropbear, initramfs-tools and cryptsetup on Debian wheezy (7.6) and I’m not able to get it working.

I am facing two issues.
– It seems the dropbear script in init-bottom directory under scripts runs before I can enter the password, hence it kills the dropbear process. If that were the only problem would be easy, because I can keep it running just commenting the line in the script where it kills the process.

– The main problem is that if, after modifying that script to keep it running, I try to connect either with private key or password, as root, it doesn’t recognise my credentials.
I have tried booting up with break=premount in the kernel like. If I start the dropbear process from there then I can connect with root, with both private key and password.

I have read some bugs in previous versions whereby it didn’t allow the access to root user, but the scripts have changed since then and it seems fine.

I am quite lost and I can’t find any similar error in the internet for Debian wheezy.

Hmm, interesting. Regarding the first problem – If the init-bottom process is killing dropbear, that means that the boot process has already moved on past prompting for a password to unlock the disk – otherwise it should be stalled there, and not get to any scripts in init-bottom until the prompting process has been killed (or successfully closed).

Do you have a graphical boot prompt (plymouth) or plain text booting? Do you have the console output to something (screen or serial port)? If you view this console somehow then do you see the local prompt for an encrypted disk password?

Regarding the second problem, I don’t have any real suggestions on that one. If you can get the boot console output as mentioned then this might shed some clues on the whole thing. Seems like both problems are related, something with your config that’s different to the ones I’ve been testing with.

Hi Angus,
My aim here is encrypt my proxmox server in hetzner, which has Debian 7.6 wheezy. But before testing in the main server I created a VM with the same debian version. Yesterday, thinking in that could be something weird going on in the VM, I tested directly in my proxmox server and the result was the same, so I am still testing in the VM.

It doesn’t have any GUI, so plymouth not installed.

What I did to debug was send some info through netcat to another server, putting the netcat command in the init-bottom. While the console is still prompting for the password I have already received the connection from netcat. And the same with the line that kills dropbear process.

As you can see I was getting and error mounting /dev/pts, since it seems it is already mounted by initramfs-tools. I commented the line where it is mounted and I don’t get that error anymore, but still the same.
I also added to modprobe blacklist the pcspkr device, as I was getting an error too just for remove errors during the boot, but as it was expected it was not related at all.

I’ll try to get an strace of the dropbear process during init-premount and when I run it manually to see if there’s any difference…

Does this work on Ubuntu 14.04.1 desktop? I would like to be able to reboot encrypted system. My whole disk is encrypted. The top comment seems to say that this won’t work. If it won’t work with Ubuntu what a bitch. Any help would be appreciated.

It seems that after the main system boots, the network interface won’t automatically change to whatever is set in /etc/network/interfaces (because the interface is seen to be already “up” it gets left alone.) To work around this you could put a line ifdown eth0; ifup eth0 in /etc/rc.local, or just keep the initramfs network settings the same as the ones in /etc/network/interfaces.”

I had severe problems getting a DHCP lease on boot. It would work right away approx. one out of ten times. The other nine times I could wait as long as I wanted, but the initramfs didn’t manage to get an ip.

The device has an ethernet port as well as a wifi card. Because I only want to connect via ethernet, I changed the initramfs networking configuration to specifically use eth0 for dhcp.
This solution which worked for me. To be precise I replaced the entry
“export IP=dhcp”
in /etc/initramfs-tools/conf.d/network_config with
“export IP=::::my-hostname:eth0:dhcp”
I’m pretty sure it’ll work just fine if you leave the hostname out tough.

Thanks for figuring this all out. I’ve been looking for something like this for some time now.

I say something *like* this, because I had in mind a slightly different flavor where the booting machine uses the dropbear client rather than its server. Upon boot, it would get a network address and then SSH out to an “authentication server” to retrieve a keyfile. Before giving up the keyfile, the authentication server would contact me via something like Duosec’s 2FA push notification. Only after I approve the request does the server “release” the keyfile and allow the rootfs to be decrypted.

The authentication server would use PKI, naturally, and would use the “command=” directive in authorized_keys to run its tasks script. It could accept some parameters, such as hostname, IP, and/or a shared-secret to log or help determine its course of action. It would then output the keyfile contents on success, which could potentially be fed directly into cryptsetup to open the LUKS partition.

The end goal would be to increase the trustability of a system when in an untrusted environment such as a remote datacenter. If I get an unexpected auth request, I have a chance to ask the remote contact what happened before I blindly unlock it. And I’m not tied to a screen to enter a password; I can approve it from anywhere as long as I have my phone with me.

Anyone have suggestions or critiques? Does it sound feasible at least?