Blog Archives

With the growing popularity of operating systems with Linux kernels that do not primarily use GNU components in the base system, this series was retitled to “GNU/Linux Crypto” in May 2017 for less ambiguity and to give proper credit to the GNU Project.

Cryptography for authentication and encryption is a complex and frequently
changing field, and for somebody new to using it, it can be hard to know where
to start. If you’re a GNU/Linux user comfortable with the terminal, but unfamiliar
with the cryptographic tools available to you on open source UNIX-like
operating systems, this series of posts aims at getting you set up with some
basic tools that will allow you to keep your own information secure, to
authenticate conveniently and safely with remote servers, and to work with
signed and encrypted files online.

I’ll be working on Debian GNU/Linux, but most of these tools should adapt well
to other open source UNIX-likes, including BSD. Please feel free to comment on
the articles with details relevant to your own implementations, or with extra
security considerations for interested readers.

As a disclaimer, I’m not myself an expert on cryptographic algorithms or key
security. If you are, and you find an error or security problem with any of my
explanations or suggestions, please let me know and I will correct it and
credit you.

Many tools that use cryptography on GNU/Linux and the internet revolve around the
Pretty Good Privacy software standard (OpenPGP). The GNU Privacy Guard
(GnuPG or GPG) is a popular free software implementation of this standard.

You can install GnuPG with its gpg(1) frontend on Debian like so:

# apt-get install gnupg

You can do a lot of very cool things with GPG, but it boils down to four
central ideas:

Generation of keypairs, randomly-generated and mathematically linked
pairs of files, one of which is kept permanently secret (the private
key) and one of which is published (the public key). This is the
basis of asymmetric key cryptography.

Managing keys, both your own public and private key, along with other
people’s public keys, so that you can verify others’ messages and files, or
encrypt them so that only those people can read them. This might include
publishing your public key to online keyservers, and getting people to sign
it to confirm that the key is really yours.

Signing files and messages with your private key to enable others to
verify that a file or message was authored or sighted by you, and not
edited in transmission over untrusted channels like the internet. The
message itself remains readable to everybody.

Encrypting files and messages with other people’s public keys, so that
only those people can decrypt and read them with their private keys. You
can also sign such messages with your own private key so that people can
verify that it was sent by you.

We’ll run through the fundamentals of each of these. We won’t concern ourselves
too much with the mathematics or algorithms behind these operations; the
Wikipedia article for asymmetric key cryptography explains this very
well for those curious for further details.

Generating a keypair

Let’s start by generating a 4096-bit RSA keypair, which should be more than
sufficient for almost everyone at the time of writing. We’ll observe a few of
the best practices recommended for the Debian developers.

Doing this on a private, up-to-date desktop machine is best, as it’s easier to
generate entropy this way. It’s still possible on an SSH-only headless server,
but you may have to resort to less cryptographically sound methods to generate
proper randomness.

Create or edit the file ~/.gnupg/gpg.conf on your system, and add the
following lines:

The expiry date is up to you. Good practice is to set an expiry date about a
year out, because as long as you have access to the private key material, you
can update the expiry date indefinitely, even if it’s already expired. For
this particular example, we’ll set an expiry date one year out:

Next, we’re prompted for some basic information to name the key. In almost all
circumstances you should use your real name, as without a real-world means to
actually verify your identity, public keys are much less useful long-term. For
the comment, you can include the key’s purpose, or your public aliases, or any
other information relevant to the key:

Key passphrase

Next, we’re prompted for a passphrase to encrypt the key, so that if it ever
falls into the wrong hands, nobody will be able to use it without knowing the
passphrase.

You need a Passphrase to protect your secret key.

Choose a sequence of random words, or possibly a unique sentence you can
easily memorise in any language, the longer the better. Don’t choose
anything that might be feasibly guessable, like proverbs or movie quotes. You
will also need to remember how you typed the passphrase exactly; I recommend
using all-lowercase and no punctuation. Wikipedia has a few guidelines
here.

You’ll need to type the passphrase twice to confirm it, and it won’t echo on
your terminal, much as if you were typing a password.

Entropy generation

Finally, the system will prompt us to generate some entropy:

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
Not enough random bytes available. Please do some other work to give
the OS a chance to collect more entropy! (Need 283 more bytes)

This step is necessary for the computer to generate sufficient random
information to ensure that the private key being generated could not feasibly
be reproduced. Moving the mouse around and using the keyboard on a desktop
system is ideal, but generating any kind of hardware activity (including
spinning disks up) should do the trick. Running expensive find(1) operations
over a filesystem (with contents that couldn’t be reasonably predicted or
guessed) helps too.

This step benefits from patience. You might find discussion online about
forcing the use of the non-blocking PRNG random device /dev/urandom instead,
using a tool like rngd(1). This definitely speeds up the process, but if
you’re going to be using your key for anything serious, I recommend actually
interacting with the computer and using hardware noise to seed the randomness
adequately, if you can.

When adequate entropy is read and the key generation is done, you’ll be
presented with some details for your master signing key pair and its encrypting
subkey pair, and the private and public keys for each are automatically added
to your keyring for use:

The directory ~/.gnupg contains the managed keys. It’s very, very important
to keep this directory private and to back it up securely, preferably to
removable media that you keep in some physically secure place. Don’t lose it!

In most contexts in GnuPG, you can refer to a key by the name of its owner, or
by its eight-digit hex ID. I prefer the latter method. Here, the short ID of my
main key is 040FE79B. While you shouldn’t use this for any actual
verification, it’s sufficiently unique that you can use it to identify
a specific key on your keyring with which you want to work.

For example, if we want to provide someone with a copy of our public key,
a friendly way to do so is to export it in ASCII format with --armor,
providing the appropriate key’s short ID:

$ gpg --armor --export 040FE79B > tom-ryder.public.asc

While you can export private keys the same way with --export-secret-key, you
should never, ever provide anyone with your private key, so this shouldn’t be
necessary.

Revocation certificate

After generating your keys, you should generate a revocation certificate:

$ gpg --output revoke.asc --gen-revoke 040FE79B
sec 4096R/040FE79B 2013-03-23 Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>
Create a revocation certificate for this key? (y/N) y
Please select the reason for the revocation:
0 = No reason specified
1 = Key has been compromised
2 = Key is superseded
3 = Key is no longer used
Q = Cancel
(Probably you want to select 1 here)
Your decision? 1
Enter an optional description; end it with an empty line:
>
Reason for revocation: Key has been compromised
(No description given)
Is this okay? (y/N) y
You need a passphrase to unlock the secret key for
user: "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"
4096-bit RSA key, ID 040FE79B, created 2013-03-23
ASCII armored output forced.
Revocation certificate created.
Please move it to a medium which you can hide away; if Mallory gets
access to this certificate he can use it to make your key unusable.
It is smart to print this certificate and store it away, just in case
your media become unreadable. But have some caution: The print system of
your machine might store the data and make it available to others!

You should store the resulting revoke.asc file somewhere safe. You can use
this certificate to revoke your key later on if the private key is ever
compromised, so that people know the key should no longer be used or trusted.
You may even like to print it out and keep a hard copy, as the output of gpg
suggests.

With the above setup done, we can proceed with some basic usage of GnuPG, as
discussed in the next article.

Subkeys

In the output of both commands, you’ll note we actually have two private and
two public keys. The sub line refers to the encryption subkey
automatically generated for you. The master key is used for cryptographic
signing, and the subkey for encryption; this is how GnuPG does things by
default with RSA keypairs.

For extra security, it might be appropriate to physically remove the master
private key from your computer, and instead use a second generated subkey for
signing files as well. This is desirable because it allows you to keep the
master key secure on some removable media (preferably with a backup), and not
loaded on your main computer in case you get compromised.

This means you can sign and encrypt files as normal with your signing subkey
and encryption subkey. If those keys ever get compromised, you can simply
revoke them and generate new ones with your uncompromised master key; everyone
who has signed your public master key or otherwise indicated they trust it will
not have to do that all over again.

With our private and public key generated and stored, we can start using a few
of GnuPG’s features to sign, verify, encrypt, and decrypt files and messages
for distribution over untrusted channels like the internet.

Signing a message or text file

We’ll start by signing a simple text file, using the --clearsign option. This
includes the signature in the message, which we can then distribute to people
to read. Here’s the contents of message.txt:

Signing and verifying binary files

For all other files, we likely need to make the signature file separate with
a detached signature:

$ gpg --armor --detach-sign archive.tar.gz

This produces a file archive.tar.gz.asc in the same directory, containing the
signature. We use --armor to make the signature in ASCII, which makes for
a longer file but easier distribution online.

In this case, both the file and the signature are required for verification;
put the signature file first when you check this:

$ gpg --verify archive.tar.gz.asc archive.tar.gz

You can use this method to verify software downloads from trusted sources, such
as the Apache HTTPD development team. First, we would download and
import all their public keys at the URL they nominate:

$ wget http://www.apache.org/dist/httpd/KEYS
$ gpg --import KEYS

We could then download an Apache HTTPD release, along with its key, from an
arbitrary mirror:

Note that the gpg output cautions that this is still not perfect assurance
that the release actually came from Jim Jagielski, because we’ve never met him
and can’t absolutely, definitely say that this is his public key. Looking him
up on public keyservers, we can however see a lot of other Apache
developers have signed his key, which looks promising, but do we know who they
are?

Despite the lack of absolute certainty, when downloading from mirrors this is
a lot better (and harder to exploit) than simply downloading without validating
or checksumming at all, given that the signature and the KEYS file were
downloaded from Apache’s own site.

You will need to decide for yourself how sure you need to be whether
a person’s public key really corresponds to them. This might extend to the
point of arranging to meet them with government-issued identification!

Encrypting a file

We can encrypt a file so that only nominated people can decrypt and read it. In
this case, we encrypt it not with our own private key, but with the recipient’s
public key. This means that they will be able to decrypt it using their own
private key.

Here’s the contents of secret-message.txt:

This is a secret message from Tom Ryder.

Now we need at least one recipient. Let’s say this message was intended for my
friend John Public. He’s given me his public key in a file called
john-public.asc on a USB drive in person; he even brought along his birth
certificate and driver’s license (which is weird, because I’ve known him since
I was four).

Now I can encrypt the message for only John to read. I like to use the 8-digit
hex code for the key for --recipient, to make sure I’ve got the right person.
You can see it in the output above, or in the output of gpg --list-keys.

If I wanted to make sure I could read the message too, I’d add my own public
key to identify myself as a recipient when I encrypt it. Then either of us will
be able to read it with our private keys (independently of the other):

These are all the basic functions of GnuPG that will be useful to most people.
We haven’t considered here submitting keys to public servers, or
participating in the web of trust; you should only look into this once
you’re happy with how your key setup is working, and are ready to publish your
key for public use.

This is appropriate for first contact with a machine, and the authentication
method is supported out of the box for most OpenSSH sshd(8) installations.

sshd(8) is a very common target for attacks, particularly automated ones;
malicious bots attempt to connect to servers listening on the standard SSH
destination port of tcp/22, as well as some common alternatives like
tcp/2222. If you enforce a strong password policy on your system, this
generally isn’t too much of a problem, particularly if only appropriate users
have shells, or if you restrict SSH connections only to certain usernames or
groups.

There are other measures to defeat automated attacks, such as employing systems
like fail2ban to reject clients who make too many spurious connection
attempts, but perhaps the most effective way of short-circuiting automated
attacks is to bypass passwords completely and instead use SSH keys,
allowing this as the only connection method to the relevant machines.

How it works

Similar to the GnuPG keys setup in the first two articles in this series, SSH
keypairs are comprised of one private key and one public key, two
cryptographically linked files. The basis of keys for authentication is that
if someone has your public key, they’re able to authenticate you by requesting
operations that you would only be able to perform with the corresponding
private key; it works similarly to cryptographic signing.

The reason this is so effective is because if you require a valid public key
to authenticate, with sufficient key length it’s effectively impossible for an
attacker to guess your authentication details; there’s no such thing as
a “common” private key to guess, so they would need to run through every
possible private key, which is not even remotely practical.

Your system’s sshd(8) may still be attacked, but if you use only public key
authentication, then you can be comfortably certain it’s effectively
impossible to brute-force your credentials. Note that this doesn’t necessarily
protect you from security problems in sshd(8) itself, however, and you will
still need to protect your private key from being hijacked or compromised,
hence the necessity of a passphrase.

All of the below assumes you have OpenSSH installed as both the client and
the server on the appropriate systems. On Debian-derived systems, these can
be installed with:

# apt-get install ssh
# apt-get install openssh-server

Both the client and server often come standard with systems (e.g. their native
OpenBSD).

Generating keys

Similar to the GnuPG setup process, we start by generating a keypair on the
machine from which we’d like to connect, using ssh-keygen(1). I’m
using 4096-bit RSA here, as it’s widely supported even on very old systems,
and should be relatively future-proof, although generating new keys if RSA
ever becomes unsafe is not hard. If you’d prefer to use the newer ECDSA
algorithm that’s the default in recent versions of OpenSSH, all of this
will still work. I’m also applying a comment for the key as an unencrypted
identifier to distinguish multiple keys if I have them. I find email addresses
work well.

First, we’re prompted for a location to which the key files should be saved. I
recommend accepting the default by pressing Enter, as using the default
location makes the next few steps easier:

Enter file in which to save the key (/home/tom/.ssh/id_rsa):

Next, we’re prompted for a passphrase, which should definitely be added to
keep the key from being used if it’s ever compromised. The same guidelines for
passphrases apply to SSH here, and you should choose a different passphrase:

Enter passphrase (empty for no passphrase):
Enter same passphrase again:

This done, the key is generated, including a pictorial representation to
recognise keys at a glance. I’ve never found this very useful, but the key
fingerprint is helpful:

The id_rsa file contains the encrypted private key, and should be kept
locked down and confidential. The id_rsa.pub file, however, contains the
public key, which can be safely distributed, in the same way as a PGP public
key.

Key-based authentication

We can now arrange to use the newly generated public key for authentication in
lieu of a password. Start by ensuring you can connect to the remote machine
with your username and password:

$ ssh remote
tom@remote's password:

Once connected, ensure that the ~/.ssh directory exists on the remote
machine, and that you don’t already have keys listed in
~/.ssh/authorized_keys, as we’re about to overwrite them:

$ mkdir -p ~/.ssh
$ chmod 0700 ~/.ssh

If this worked, close the connection (exit or Ctrl-D) to return to your
local machine’s shell, and copy your public key onto the remote machine with
scp(1):

Note that there’s a tool included in recent versions of OpenSSH that does this
for you called ssh-copy-id(1), but it’s good to have some idea of
what it’s doing in the background.

With this done, your next connection attempt to the remote host should prompt
you for your passphrase, rather than your password:

$ ssh remote
Enter passphrase for key '/home/tom/.ssh/id_rsa':

Advantages

At first, it may not seem like you’ve done much useful here. After all, you
still have to type in something to connect each time. From a security
perspective, the first major advantage to this method is that neither your
password, nor your passphrase, nor your private key are ever transmitted to
the server to which you’re connecting; authentication is done purely based on
the public-private key pair, decrypted by your passphrase.

This means that if the machine you’re connecting to were compromised, or your
DNS had been poisoned, or some similar attack tricked you into connecting to a
fake SSH daemon designed to collect credentials, your private key and your
password remain safe.

The second advantage comes with turning off password authentication entirely on
the host machine, once all its users have switched to public key authentication
only. This is done with the following settings in sshd_config(5),
usually in /etc/ssh/sshd_config on the remote server:

PubkeyAuthentication yes
ChallengeResponseAuthentication no
PasswordAuthentication no

Restart the SSH server after these are applied:

$ sudo /etc/init.d/ssh restart

You should then no longer be able to connect via passwords at all, only by
private keys, which as mentioned above are effectively (though not literally)
impossible to brute-force. In order to connect to the server as you, an
attacker would not only need to know your passphrase, but also have access to
your private key, making things significantly harder.

Using public key authentication also allows sshd(8) some finer-grained
control over authentication, such as which hosts can connect with which keys,
whether they can execute TCP or X11 tunnels, and (to an extent) which commands
they can run once connected. See the manual page for
authorized_keys(5) to take a look at some examples.

Finally, there’s a major usability advantage in using SSH keys for
authentication with agents, which we’ll discuss in the next article.

Host keys and fingerprints

SSH connection should ideally be a two-way authentication process. Just as the
server to which you’re connecting needs to be sure who you are, you need to be
sure that the host you’re connecting to is the one you expect. With tunnelling,
firewalls, DNS poisoning, NAT, hacked systems, and various other tricks, it’s
appropriate to be careful that you’re connecting to the right systems. This is
where OpenSSH’s host key system comes into play.

The first time you connect to a new server, you should see prompts like this:

$ ssh newremote
The authenticity of host 'newremote (192.168.0.65)' can't be established.
RSA key fingerprint is f4:4b:f4:8c:c5:50:f6:c8:d3:b2:e9:14:68:86:b5:7b.
Are you sure you want to continue connecting (yes/no)?

A lot of administrators turn these off; don’t! They are very important.

The key fingerprint is a relatively short hash for the host key used by
OpenSSH on that server. It’s verified by your SSH client, and can’t easily be
faked. If you’re connecting to a new server, it’s appropriate to check the host
key fingerprint matches the one you see on first connection attempt, or to ask
the system’s administrator to do so for you.

The host key’s fingerprint can be checked on the SSH server with a call to
ssh-keygen(1):

If you can’t check the host key yourself, have the administrator send it to you
over a secure, trusted channel, such as in person or via a PGP signed message.
If the colon-delimited SSH fingerprint is not exactly the same, then you might
be the victim of someone attempting to spoof your connection!

This is definitely overkill for new virtual machines and probably new machines
on a trusted LAN, but for machines accessed over the public internet, it’s a
very prudent practice.

Similarly, ssh(1) by default keeps a record of the host keys for hosts,
which is why when a different host key is presented on a connection attempt, it
warns you:

$ ssh newremote
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
d7:06:51:16:80:f6:32:b4:35:7c:53:8d:5a:49:69:ec
Please contact your system administrator.
Add correct host key in /home/tom/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/tom/.ssh/known_hosts:22
RSA host key for newremote has changed and you have requested strict checking.

Again, this is something ssh(1) users often turn off, which is pretty risky,
especially if you are using password authentication and hence might send your
password to a malicious or compromised server!

Now that we have both GnuPG and SSH securely set up, we’re able to encrypt,
decrypt, sign, and verify messages, and securely authenticate to remote servers
without any risk of exposing passwords and with effectively zero possibility of
a brute-force attack. This is all great, but there’s still one more weak link
in the chain with which to deal — our passphrases.

If you use these tools often, typing your passphrase for most operations can
get annoying. It may be tempting to either include a means of automating the
passphrase entry, or not to use a passphrase at all, leaving your private key
unencrypted. As security-conscious users, we definitely want to avoid the
latter in case our private key file ever gets stolen, which is where the
concepts of agents for both SSH and GnuPG comes into play.

An agent is a daemon designed to streamline the process of using decrypted
private keys by storing the details securely in memory, ideally for a limited
period of time. What this allows you do with both SSH and GnuPG is to type your
passphrase just once, and subsequent uses that require the unencrypted private
key are managed by the agent.

In this article, we’ll go through the basics of agent setup for both SSH and
GnuPG. Once we know how they work, we’ll then introduce a convenient tool to
start both of them and manage them for us easily.

SSH agents

The ssh-agent(1) program comes as part of the OpenSSH suite. It
can be run in two modes, either as a parent process, or daemonized into the
background. We’ll discuss the latter method, as it’s more commonly used and
more flexible.

Setup

When we run ssh-agent(1) for the first time, its behavior is curious; it
appears to do nothing except spit out some cryptic shell script:

However, we can see that the daemon is running with the PID it mentions:

$ ps 28882
PID TTY STAT TIME COMMAND
28882 ? Ss 0:00 ssh-agent

So if it’s running fine, what’s with all the shell script it outputs? Why
doesn’t it just run that for us?

The answer is an interesting workaround to a stricture of the Unix process
model; specifically, a process cannot modify its parent environment. The
variables SSH_AUTH_SOCK and SSH_AGENT_PID are designed to allow programs
like ssh(1) to find the agent so it can communicate with it, so we definitely
need them set. However, if ssh-agent(1) were to set these variables itself,
it would only apply for its own process, not the shell where we called it.

Therefore, not only do we need to run ssh-agent(1), we need to execute the
code it outputs so the variables get assigned in our shell. A good method of
doing this in Bash is using eval and command substitution with $(...):

$ eval "$(ssh-agent)"
Agent 3954

If we run this, we can see that not only is ssh-agent(1) running, we have two
new variables in our environment identifying its socket path and process ID:

With this done, the agent is ready, and we can start using it to manage our
keys for us.

Usage

The next step is to load our keys into the agent with ssh-add(1). Pass
this program the full path to the private key you would like to use with the
agent. This is likely either ~/.ssh/id_rsa or ~/.ssh/id_dsa:

The default is to maintain the keys permanently, until the agent is stopped or
the keys are explicitly removed one-by-one with ssh-add -d <keyfile> or all
at once with ssh-add -D. For the cautious, you can set a time limit in
seconds with ssh-add -t. For example, to have ssh-add forget about your
keys after two hours, you might use:

$ ssh-add -t 7200 ~/.ssh/id_rsa

To kill the agent completely, you can use ssh-agent -k, again with an eval
$(...) wrapper:

$ eval "$(ssh-agent -k)"
Agent pid 4501 killed

You may like to consider adding this to ~/.bash_logout or a similar script to
get rid of the running agent after you’re done with your session.

Permanent setup

If you like this and find it makes your key management more convenient, it
makes sense to put it into a startup script like ~/.bash_profile. This way,
the agent will be started for each login shell, and we will be able to
communicate with it from any subshell (xterm, screen, or an appropriately
configured tmux):

eval "$(ssh-agent)"
ssh-add ~/.ssh/id_rsa

On our next TTY login, we should be prompted for a passphrase, and from there
be able to connect to any machine using the keys managed by the agent:

We thus have to type our passphrase only once per login session, and can
connect to all of the servers to which our keys confer access … very
convenient!

GnuPG Agents

Just like ssh-agent(1), there exists an agent for managing GnuPG keys too,
called gpg-agent(1). Its behavior is very similar. On Debian-derived
systems, it can be installed as the gnupg-agent package. You should
also install a pinentry program; as we’re focussing on learning the nuts and
bolts on the command line here, we’ll use pinentry-curses(1) for a
console-based passphrase prompt:

# apt-get install gnupg-agent pinentry-curses

Setup

We’ll start the agent using the same eval $(...) trick we learned with
ssh-agent:

$ eval "$(gpg-agent --daemon)"

We can verify that the agent is running in the background with the given PID,
and that we have a new environment variable:

The agent has thus cached the private key for us, making it much easier to
perform a series of operations with it. The default timeout is 10 minutes, but
you can change this with the default-cache-ttl and max-cache-ttl settings
in ~/.gnupg/gpg-agent.conf. For example, to retain any private key for one
hour after its last use and a maximum of two hours from its first use, we could
write:

default-cache-ttl 3600
max-cache-ttl 7200

Changing these values will require prompting the agent to reload:

$ gpg-connect-agent <<<RELOADAGENT
OK

Permanent setup

Just like ssh-agent(1), an ideal place for gpg-agent(1)‘s startup lines is
in a login shell setup script like ~/.bash_profile:

eval "$(gpg-agent --daemon)"

The agent will be started, and all of its environment variables will be set and
inherited by all subshells, just as with ssh-agent.

If you’re using the console PIN entry tool, you should also add this to end of
your interactive shell startup script. This should be ~/.bashrc for Bash on
Linux; you may need to put it in ~/.bashrc on Mac OS X.

export GPG_TTY=$(tty)

Keychain

To manage both ssh-agent(1) and gpg-agent(1) effectively, a tool called
keychain(1) is available. It provides a simple way to start both
agents with one command, including loading keys at startup, and also prevents
running either agent twice, picking up on agents started elsewhere on the
system. Because desktop environments are often configured to start one or both
agents for users, it makes sense to re-use them where possible, at which
keychain(1) excels.

With keychain installed, we can start both agents with just one command in
~/.bash_profile:

eval "$(keychain --eval)"

We can optionally include the filenames of SSH keys in ~/.ssh or the hex IDs
of GnuPG keys as arguments to prompt loading the private key (including
requesting the passphrase) at startup:

eval "$(keychain --eval id_rsa 0x77BB8872)"

If this program is available to you, then I highly recommend this; managing
agents and environments can be fiddly, and keychain(1) does all the hard work
for you in this regard so you don’t have to worry about whether an agent is
available to you in your particular context. Check out the project’s
homepage for more information about the tool.

It’s now becoming more widely known that using guessable passwords or using the
same password for more than one account is a serious security risk, because an
attacker able to control one account (such as an email account) can do a lot of
damage. If an attacker gets the hash of your password from some web service,
you want to be assured that the hash will be very difficult to reverse, and
even if it can be reversed, that it’s unique and won’t give them access to any
of your other accounts.

This growing awareness has contributed to the popularity of password
managers, tools designed to securely generate, store, and retrieve passwords,
encrypted with a master password or passphrase. In some cases these are locally
stored, such as KeePass, and in others they are stored on a web service,
such as LastPass. Both are good tools, and work well with GNU/Linux. I
personally have some reservations about LastPass as I don’t want my passwords
stored on a third party service, and I don’t trust JavaScript encryption.

Interestingly, because we now have a tidy GnuPG setup to handle the encryption
ourselves, another option is the pass(1) tool, billing itself as “the
standard UNIX password manager”. It’s little more than a shell script and some
bash(1) completions wrapped around existing tools like git(1),
gpg2(1), pwgen(1), tree(1), and xclip(1),
and your choice of $EDITOR. If you’re not already invested in an existing
password management method, you might find this a good first application of
your new cryptography setup, and a great minimal approach to secure password
storage accessible from the command line (and therefore SSH).

On Debian-derived systems, it’s available as part of the pass package:

# apt-get install pass

This includes a manual:

$ man pass

Instructions for installing on other operating systems are also available on
the site. Releases are also available for download, and a link to the
development repository. If you use this, make sure you have the required
tools outlined above installed as well, although xclip(1) is only needed if
you run the X Windows system.

Setup

We can get an overview of what pass(1) can do by invoking it with no
arguments:

$ pass

To start, we’ll initialize our password store. For your own passwords, you will
want to do this as your own user rather than root. Because pass(1) uses
GnuPG for its encryption, we also need to tell it the ID of the appropriate key
to use. Remember, you can find this eight-digit hex code by typing gpg
--list-secret-keys. A unique string identifying your private key such as your
name or email address may also work.

pass(1) uses pwgen(1) for this password generation. In each case, the
password is automatically inserted into the password store for you.

If we need to change an existing password, we can either overwrite it with
insert again, or use the edit operation to invoke our choice of $EDITOR:

$ pass edit google.com/gmail/example@gmail.com

If you do this, you may like to be careful that your editor is not configured
to keep backups or swap files in plain text of documents it edits in temporary
directories or memory filesystems. If you’re using Vim, I wrote a plugin
in an attempt to solve this problem.

Note that adding or overwriting passwords does not require your passphrase;
only retrieval and editing does, consistent with how GnuPG normally works.

Retrieval

This password can now be retrieved and echoed onto the command line given the
appropriate passphrase:

In each case, note that if you have the bash completion installed and working,
you should be able to complete the full path to the passwords with Tab, just
as if you were directly browsing a directory hierarchy.

Deletion

If we no longer need the password, we can remove it with pass rm:

$ pass rm google.com/gmail/example@gmail.com
Are you sure you would like to delete google.com/gmail/example@gmail.com? [y/N] y
removed ‘/home/tom/.password-store/google.com/gmail/example@gmail.com.gpg’

Backups

Because the password files are all encrypted only to your GnuPG key, you can
relatively safely back up the store on remote and third-party sites simply by
copying the ~/.password-store directory. If the filenames themselves contain
sensitive information, such as private usernames or sites, you might like to
back up an encrypted tarball of the store instead:

An encrypted password storage is well and good, but now that we have a working
GnuPG setup, we should consider using PGP for what it was originally designed:
email messages. To do this, we’ll be using Mutt.

Mutt is a console-based mail user agent, or MUA, designed chiefly for managing
and reading mail. Unlike mailer programs like Thunderbird, it was not
designed to be a POP3/IMAP client, or an SMTP agent, although versions in
recent years do include this functionality; these are tasks done by programs
like Getmail and MSMTP.

If like many people, you’re using Gmail, this works very well with POP3/IMAP
and SMTP, enabling you to compose email messages in plain text with your choice
editor in a terminal window, in a highly configurable environment, and doing
your own email encryption for any sensitive communications in such a way that
even your email provider can’t read it.

General usage of Mutt and setup for Gmail users is not covered in detail here,
although it may be the subject of a later article. For now, there are many
excellent articles on the basics of a Mutt setup. If you’re interested
in the setup for other GNU/Linux mail clients like Claws or Thunderbird,
Cory Sadowski has a very good article walking you through that, among
other privacy settings relevant to both GNU/Linux and Windows.

All of the below is assuming you already have a GnuPG keypair ready,
with gpg-agent(1) running in the background to manage your keys.

Background

Most of the PGP setup guides for Mutt you can find online are quite old,
and they usually suggest a lot of lines of .muttrc configuration to interface
directly with the gpg command, with a myriad of options and some byzantine
variable substitution:

I’m all for the Unix philosophy of using programs together, but this is just
too much. It’s a fickle setup that’s very hard to work with, and it requires
too much understanding of the gpg(1) frontend to use and edit sensibly. After
all, we want to end up with a setup that we understand reasonably well.

So, throw all that away; we’re going to use GPGME instead. The above is
exactly the problem that this library is designed to solve; it’s a library to
which applications can link to streamline the usage of GnuPG functions,
including interfacing with agents. We can replace all of the above with this:

set crypt_use_gpgme = yes

Installation

If you have Mutt installed, odds are it already has a GPGME interface. You can
check if your current version of Mutt has GPGME powers by looking at the mutt
-v version output. Here’s the output of mine, using the packaged Mutt from
Debian GNU/Linux, which does have GPGME support:

Remember, it’s your responsibility to decide how much you trust this public
key; normally it’s best practice that you actually meet someone in person to
exchange written key fingerprints in order to be completely sure that the key
corresponds to that user.

If you don’t have anyone else you know using PGP to communicate with, you can
send me a message encrypted with my public key 0xC14286EA77BB8872 to
tom@sanctum.geek.nz. If you send or link me your public key in your message,
then I’ll reply to you with a message encrypted with your public key so you can
check everything’s working.

EDIT 2017: Yes, this offer is still good; you can still email me if you want
to test your new setup; I will still reply to you!

Back in Mutt, begin composing a message with m. Enter the recipient and
subject as normal, and compose your message. When you’ve finished writing and
save and quit $EDITOR, and your message is in the Compose screen waiting to
send, press p to bring up the PGP menu at the bottom:

PGP (e)ncrypt, (s)ign, sign (a)s, (b)oth, s/(m)ime or (c)lear?

We’ll press b to both sign and encrypt the message.

If you want to be able to read the message after sending, then you’ll need to
arrange to encrypt it with your key as well as the recipient’s. I find the
cleanest way to do this is to add your address to the Bcc: header with b.
You can also set this as a default with the following line in
~/.gnupg/gpg.conf, where 0x1234ABCD is the short ID of your own key:

encrypt-to 0x1234ABCD

When you send the message with y, you might need to specify which key you
want to use for each recipient, if you don’t have a unique key on your keychain
with your recipient’s email address.

When you send, you should be prompted for your passphrase by your PIN entry
program, unless your agent is already holding the key for you. This is needed
in order to sign the message. When you’ve provided this, the message will be
sent, and if you included yourself in the Bcc: field, you should be able to
read it in your sent mail, with some headers showing the PGP information
(whether the message was signed, encrypted, or both):

Your recipient will be able to decrypt the message in their mail user agent
with their private key, and nobody else but the two of you will be able to read
it. Note that this works for any number of recipients, as long as you have a
public key for each of them.

Keep in mind that the metadata of the message, such as the sender and recipient
name and address, date and time it was sent, and (importantly) the subject, are
sent in plain text. Only the body of the message (including attachments) is
encrypted.

Useful extras

With GPGME, Mutt tries to use the first secret key available to it in its
private keychain. If you want to use some other specific keypair for signing
messages, you can specify that with with the pgp_sign_as option in .muttrc:

set pgp_sign_as = 0x9876FEDC

If you’d like to automatically sign all of your outgoing mail, you can set the
crypt_autosign option:

set crypt_autosign = yes

The first batch of options we set earlier will already automatically sign
and/or encrypt messages in responses to messages doing either/both.

If you’d like to include a link to your PGP key in the headers to each message,
you can add a custom header with my_hdr:

All of this combines with Mutt’s extensive speed and high-powered configuration
to make Mutt a very capable and convenient PGP mail client. As always, the more
people you know using PGP, and the more public keys you have, the more useful
this will be.

While having local backups for quick restores is important, such as on a USB
disk or spare hard drive, it’s equally important to have a backup offsite from
which you can restore your important documents if, for example, your office was
burgled or burned down, losing both your workstation and backup media.

The easiest way to do this for most people is with a storage provider, offering
convenient access to bulk storage of suitable size maintained on another
company’s systems for a relatively modest price or even for free, such as the
Ubuntu One service, or Microsoft’s offering, Skydrive. The best
storage providers will also encrypt the data on their own servers, whether or
not they have access.

Trusting a company with all your data and the encryption thereof is risky,
particularly given recent revelations of corporate collusion with the
NSA, and privacy-conscious users should prefer the security of encrypting
the backups before they go up onto the provider’s servers. The provider may
implement closed and/or symmetric encryption mechanisms of their own, which may
or may not be trustworthy. For very strong personal encryption, as established,
we can use our GnuPG setup to encrypt files before we put them up there:

The problem with encrypting whole files before we put them up for storage is
that for even modestly sized data, performing entire backups and uploading all
of the files together every time can cost a lot of bandwidth. Similarly, we’d
like to be able to restore our personal files as they were on a specific date,
in case of bad backups or accidental deletion, but without storing every file
on every backup day, which may end up requiring far too much space.

Incremental backups

Normally, the solution is to use an incremental backup system, meaning after
first uploading your files in their entirety to the backup system, successive
backups upload only the changes, storing them in a retrievable and
space-efficient format. Systems like Dirvish, a free Perl frontend to
rsync(1), allow this.

Unfortunately, Dirvish doesn’t encrypt the files or changesets it stores.
What’s needed is an incremental backup solution that efficiently calculates and
stores changes in files on a remote server, and also encrypts them.
Duplicity, a Python tool built around librsync, excels at this, and
can use our GnuPG asymmetric key setup for the file encryption. It’s available
in Debian-derived systems in the duplicity package. Note that, as
before, a GnuPG key setup with an agent is required for this to work.

Usage

We can get an idea of how duplicity(1) works by asking it to start a
backup vault on our local machine. It uses much the same sourcedestination
argument as tools like rsync or scp:

It’s important to specify --encrypt-key, because otherwise duplicity(1)
will use symmetric encryption with a passphrase rather than a public key, which
is considerably less secure. Specify the email address corresponding to the
public keypair you would like to use for the encryption.

This performs a full encrypted backup of the directory, returning the following
output:

You’ll note you were not prompted for your passphrase to do this. Remember,
encrypting files with your public key does not require a passphrase; the whole
idea is that anyone can encrypt using your key without needing your permission.

Checking the created directory docsbackup, we find three new files within it,
all three of them encrypted:

The vol1.difftar.gpg file contains the actual data stored; the other two
files contain metadata about the backup’s contents, for use to calculate
differences the next time the backup runs.

If we make a small change to a file in the directory being backed up, and run
the same command again, we note that the backup has been performed
incrementally, and only the changes (the new file) have been saved:

Note that the new files have the prefix duplicity-inc- or duplicity-new-,
denoting them as incremental backups and not full ones.

Note that in order to keep track of what files have already been backed up,
duplicity(1) stores metadata in ~/.cache/duplicity, as well as storing them
along with the backup. This allows us to let our backup processes run
unattended, rather than having to put in our passphrase to read the metadata on
the remote server before performing an incremental backup. Of course, if we
lose our cached files, that’s OK; we can read the ones out of the backup vault
by supplying our passphrase on request for decryption.

Remote backups

If you have SSH or even just SCP/SFTP access to your storage provider’s
servers, not much has to change to make duplicity(1) store the files up there
instead:

Your backups will then be sent over an SSH link to the directory docsbackup
on the system backup.example.com, with username user. In this way, not only
is all the data protected in transmission, it’s stored encrypted on the remote
server; it never sees your plaintext data. All anyone with access to your
backups can see is their approximate size, the dates they were made, and (if
you publish your public key) the user ID on the GnuPG key used to encrypt them.

The duplicity(1) frontend supports other methods of uploading to different
servers, too, including the boto backend for S3 Amazon Web Services, the
gdocs backend for Google Docs, and httplib2 or oauthlib for Ubuntu One.

If you like, you can also sign your backups to make sure they haven’t been
tampered with at the time of restoration, by changing --encrypt-key to
--encrypt-sign-key. Note that this will require your passphrase.

Restoring

Restoring from a duplicity(1) backup volume is much the same, but with the
arguments reversed:

Note that this time you are asked for your passphrase. This is because
restoring the backup requires decrypting the data and possibly the signatures
in the backup vault. After doing this, the complete set of documents from the
time of your most recent incremental backup will be available in docsrestore.

Using this incremental system also allows you to restore your data in the state
in the last backup before a given time. For example, to retrieve my
~/Documents directory as it was three days ago, I might run this:

Automating

You should run your first full backup interactively to make sure it’s doing
exactly what you need, but once you’re confident that everything is working
correctly, you can set up a simple Bash script to run incremental backups for
you. Here’s an example script, saved in $HOME/.local/bin/backup-remote:

The line with keychain is optional, but will be necessary if you’re using an
SSH key with a passphrase on it; you’ll also need to have authenticated with
ssh-agent at least once. See the earlier article on SSH/GPG agents for
details on this setup.

The following line would run this script every morning, beginning at 6.00am:

0 6 * * * ~/.local/bin/backup-remote

Tips

A few general best practices apply to this, consistent with the Tao of
Backup:

Check that your backups completed; either have the output of the cron
script mailed to you, or log it to a file that you check at least
occasionally to make sure your backups are working. I highly recommend
using an email message, and including error output:

MAILTO=you@example.com
0 6 * * * ~/.local/bin/backup-remote 2>&1

Run backups to your local servers too; this might prevent your backup
provider from reading your files, but it won’t save them from being
accidentally deleted.

Don’t forget to occasionally test-restore your backups to make sure they’re
working correctly. It’s also wise to use duplicity verify on them
occasionally, particularly if you don’t back up every day:

This incremental system means that you’ll likely only have to make full
backups once, so you should back up too much data rather than too little;
if you can spare the bandwidth and have the space, backing up your entire
computer isn’t really that extreme.

Try not to depend too much on your remote backups; see them as a last
resort, and work securely and with local backups as much as you can.
Certainly, never rely on backups as a version control system; use Git for
that.

GnuPG provides us with a means to securely encrypt individual files on a
filesystem, but for really high-security information or environments, it may be
appropriate to encrypt an entire disk, to mitigate problems such as caching
sensitive files in plaintext. The GNU/Linux kernel
includes its own disk encryption solution in the kernel, dm-crypt. This can
be leveraged with a low-end tool called cryptsetup, or more easily with LUKS,
the Linux Unified Key Setup, implementing strong cryptography with
passphrases or keyfiles.

In this example, we’ll demonstrate how this can work to encrypt a USB drive,
which is a good method for securely storing really sensitive data such as PGP
master keys that’s only needed occasionally, rather than leaving it always
mounted on a networked device. Be aware that this erases any existing files
on the drive.

Installation

The cryptographic tools used by dm-crypt and LUKS are built-in to Linux
kernels after 2.6, but you may have to install a package to get access to the
cryptsetup frontend. On Debian-derived systems, it’s available in
cryptsetup:

# apt-get install cryptsetup

On RPM-based systems like Fedora or CentOS, the package has the same name,
cryptsetup:

# yum install cryptsetup

Creating the volume

After identifying the block device on which we want the encrypted filesystem,
for example /dev/sdc1, we can erase any existing contents using a call to
wipefs:

# wipefs -a /dev/sdc1

Alternatively, we can zero out the whole disk, if we want to completely
overwrite any trace of the data previously on the disk; this can take a long
time for large volumes:

# cat /dev/zero >/dev/sdc1

If you don’t have a USB drive to hand, but would still like to try this out,
you can use a loop block device in a file. For example, to create a 100MB loop
device:

You can then follow the rest of this guide using /dev/loop0 as the raw block
device in place of /dev/sdc1. In the above output, losetup -f returns the
first available loop device for use.

Setting up a LUKS container on the block device is then done as follows,
providing a passphrase of decent strength; as always, the longer the better.
Ideally, you should not use the same passphrase as your GnuPG or SSH keys.

This creates an abstracted encryption container on the disk, which can be
opened by providing the appropriate passphrase. A virtual mapped device is
then provided that encrypts all data written to it transparently, with the
encrypted data written to the disk.

Using the mapped device

We can open the mapped device using cryptsetup luksOpen, which will prompt us
for the passphrase:

# cryptsetup luksOpen /dev/sdc1 secret

The last argument here is the filename for the block device to appear under
/dev/mapper; this example provides /dev/mapper/secret.

With this done, the block device on /dev/mapper/secret can now be used in the
same way as any other block device; all of the disk operations are abstracted
through encryption operations. You’ll probably want to create a filesystem on
it; in this case, I’m creating an ext4 filesystem:

We can then mount the device as normal, and data put into the newly created
filesytem will be transparently encrypted:

# mkdir -p /mnt/secret
# mount /dev/mapper/secret /mnt/secret

For example, we could store a private GnuPG key on it:

# cp -prv /home/tom/.gnupg/secring.gpg /mnt/secret

Information about the device

We can get some information about the LUKS container and the specifics of its
encryption using luksDump on the underlying block device. This shows us the
encryption method used, in this case aes-xts-plain64.

Unmounting the device

When finished with the data on the device, we should both unmount any
filesystem on it, and also close the mapped device so that the passphrase is
required to re-open it:

# umount /mnt/secret
# cryptsetup luksClose /dev/mapper/secret

If the data is a removable device, you should also consider physically removing
the media from the machine and placing it in some secure location.

This post only scratches the surface of LUKS functionality; many more things
are possible with the system, including automatic mounting of encrypted
filesystems and the use of stored keyfiles instead of typed passphrases. The
FAQ for cryptsetup contains a great deal of information, including
some treatment of data recovery, and the Arch Wiki has an exhaustive
page on various ways of using LUKS securely.

While this series was being written, from June 2013, Edward Snowden
began leaking top-secret documents from the United States National Security
Agency, showing that the agency was capable of Internet surveillance on
a massive scale with the PRISM surveillance system and with the XKeyscore
interface into their amassed data. The fact that covert government surveillance
was possible and was taking place does not come as particularly surprising news
to network engineers and conspiracy theorists, but the revelations have finally
given the general, non-technical public an idea of how badly the proprietary
systems around which they have built much of their digital lives can be used to
harm them and compromise their privacy.

Concerned people in the United States will be only too aware of how the secret
abuse of power to exercise this surveillance and the failed motions to curtail
it by the United States Congress has dented their trust in their own
government. However, the leaks’ implications are international as well. The
foreign intelligence agency in my own country of New Zealand, the Government
Communications Security Bureau, was earlier this year accused of illegally
spying on New Zealand citizens, and diplomatic cables from WikiLeaks show
the GCSB is potentially already cooperating with the NSA. In spite of
this, new legislation is set to extend the GCSB’s powers, despite independent
reviews condemning the bill from both a legal and human rights
perspective, even after amendments. The scandal and the anger over surveillance
abuse extends to the United Kingdom, Germany, Sweden, and
many other countries.

I do hold out some hope for the efforts such as the Electronic Frontier
Foundation’s class action suit to curtail the surveillance or at the
very least to register the public’s anger about this unwarranted intrusion into
private lives. However I am concerned not just by the possibility of the rise
of a global surveillance state, but by the implications this has for the right
to secure communications using cryptography for authentication and encryption.

It’s no secret that cryptography and encryption presents a problem to the
NSA’s surveillance systems, and that they expend a great deal of effort
in attempting to circumvent it, including demanding private keys from
businesses for applications like HTTPS. My concern is this: If it becomes
publically accepted that governments spy warrantlessly on international
networks and that this is justified or necessary, then we may reach a point
where the legality of the general public’s use of cryptography itself may again
be called into question.

Computing professionals of my generation likely did not begin their careers
until after the United States’ cryptographic export controls were relaxed in
1999, perhaps prompting us to take for granted the availability of algorithms
like RSA and AES with high key sizes for cryptographic purposes. A world where
a government agency would actively attempt to curtail the use of such
technology may seem very far-fetched to us — perhaps less so to those who
remember that Pretty Good Privacy was a radical new idea that caused its
activist creator Phil Zimmermann real legal trouble.

I believe that computing enthusiasts and users of free software operating
systems, not just cryptographic experts, are in a special position to assist
their concerned friends and family with defending their online privacy and
securing their communications, and that if we value both freedom and security
of information, then we in fact have a responsibility to do
so. I believe that people need to be aware of not just the implications of
massive surveillance on a global scale, but also how to exercise their rights
to fight against it. If the legality of cryptography is ever called into
question again as the result of its impeding warrantless surveillance, then its
pervasiveness and the public’s insistence on its free availability should make
restricting its use not just impractical, but unthinkable.