Repository: All data produced during a backup is sent to and stored in
a repository in a structured form, for example in a file system
hierarchy with several subdirectories. A repository implementation must
be able to fulfill a number of operations, e.g. list the contents.

Blob: A Blob combines a number of data bytes with identifying
information like the SHA-256 hash of the data and its length.

Pack: A Pack combines one or more Blobs, e.g. in a single file.

Snapshot: A Snapshot stands for the state of a file or directory that
has been backed up at some point in time. The state here means the
content and meta data like the name and modification time for the file
or the directory and its contents.

Storage ID: A storage ID is the SHA-256 hash of the content stored in
the repository. This ID is required in order to load the file from the
repository.

All data is stored in a restic repository. A repository is able to store
data of several different types, which can later be requested based on
an ID. This so-called “storage ID” is the SHA-256 hash of the content of
a file. All files in a repository are only written once and never
modified afterwards. This allows accessing and even writing to the
repository with multiple clients in parallel. Only the delete operation
removes data from the repository.

Repositories consist of several directories and a top-level file called
config. For all other files stored in the repository, the name for
the file is the lower case hexadecimal representation of the storage ID,
which is the SHA-256 hash of the file’s contents. This allows for easy
verification of files for accidental modifications, like disk read
errors, by simply running the program sha256sum on the file and
comparing its output to the file name. If the prefix of a filename is
unique amongst all the other files in the same directory, the prefix may
be used instead of the complete filename.

Apart from the files stored within the keys directory, all files are
encrypted with AES-256 in counter mode (CTR). The integrity of the
encrypted data is secured by a Poly1305-AES message authentication code
(sometimes also referred to as a “signature”).

In the first 16 bytes of each encrypted file the initialisation vector
(IV) is stored. It is followed by the encrypted data and completed by
the 16 byte MAC. The format is: IV||CIPHERTEXT||MAC. The
complete encryption overhead is 32 bytes. For each file, a new random IV
is selected.

The file config is encrypted this way and contains a JSON document
like the following:

After decryption, restic first checks that the version field contains a
version number that it understands, otherwise it aborts. At the moment,
the version is expected to be 1. The field id holds a unique ID
which consists of 32 random bytes, encoded in hexadecimal. This uniquely
identifies the repository, regardless if it is accessed via SFTP or
locally. The field chunker_polynomial contains a parameter that is
used for splitting large files into smaller chunks (see below).

A local repository can be initialized with the resticinit command,
e.g.:

$ restic -r /tmp/restic-repo init

The local and sftp backends will auto-detect and accept all layouts described
in the following sections, so that remote repositories mounted locally e.g. via
fuse can be accessed. The layout auto-detection can be overridden by specifying
the option -olocal.layout=default, valid values are default and
s3legacy. The option for the sftp backend is named sftp.layout, for the
s3 backend s3.layout.

Unfortunately during development the AWS S3 backend uses slightly different
paths (directory names use singular instead of plural for key,
lock, and snapshot files), and the data files are stored directly below
the data directory. The S3 Legacy repository layout looks like this:

All files in the repository except Key and Pack files just contain raw
data, stored as IV||Ciphertext||MAC. Pack files may contain one
or more Blobs of data.

A Pack’s structure is as follows:

EncryptedBlob1||...||EncryptedBlobN||EncryptedHeader||Header_Length

At the end of the Pack file is a header, which describes the content.
The header is encrypted and authenticated. Header_Length is the
length of the encrypted header encoded as a four byte integer in
little-endian encoding. Placing the header at the end of a file allows
writing the blobs in a continuous stream as soon as they are read during
the backup phase. This reduces code complexity and avoids having to
re-write a file once the pack is complete and the content and length of
the header is known.

All the blobs (EncryptedBlob1, EncryptedBlobN etc.) are
authenticated and encrypted independently. This enables repository
reorganisation without having to touch the encrypted Blobs. In addition
it also allows efficient indexing, for only the header needs to be read
in order to find out which Blobs are contained in the Pack. Since the
header is authenticated, authenticity of the header can be checked
without having to read the complete Pack.

This is enough to calculate the offsets for all the Blobs in the Pack.
Length is the length of a Blob as a four byte integer in little-endian
format. The type field is a one byte field and labels the content of a
blob according to the following table:

Type

Meaning

0

data

1

tree

All other types are invalid, more types may be added in the future.

For reconstructing the index or parsing a pack without an index, first
the last four bytes must be read in order to find the length of the
header. Afterwards, the header can be read and parsed, which yields all
plaintext hashes, types, offsets and lengths of all included blobs.

Index files contain information about Data and Tree Blobs and the Packs
they are contained in and store this information in the repository. When
the local cached index is not accessible any more, the index files can
be downloaded and used to reconstruct the index. The files are encrypted
and authenticated like Data and Tree Blobs, so the outer structure is
IV||Ciphertext||MAC again. The plaintext consists of a JSON
document like the following:

This JSON document lists Packs and the blobs contained therein. In this
example, the Pack 73d04e61 contains two data Blobs and one Tree
blob, the plaintext hashes are listed afterwards.

The field supersedes lists the storage IDs of index files that have
been replaced with the current index file. This happens when index files
are repacked, for example when old snapshots are removed and Packs are
recombined.

There may be an arbitrary number of index files, containing information
on non-disjoint sets of Packs. The number of packs described in a single
file is chosen so that the file size is kept below 8 MiB.

All data stored by restic in the repository is encrypted with AES-256 in
counter mode and authenticated using Poly1305-AES. For encrypting new
data first 16 bytes are read from a cryptographically secure
pseudorandom number generator as a random nonce. This is used both as
the IV for counter mode and the nonce for Poly1305. This operation needs
three keys: A 32 byte for AES-256 for encryption, a 16 byte AES key and
a 16 byte key for Poly1305. For details see the original paper The
Poly1305-AES message-authentication
code by Dan Bernstein.
The data is then encrypted with AES-256 and afterwards a message
authentication code (MAC) is computed over the ciphertext, everything is
then stored as IV || CIPHERTEXT || MAC.

The directory keys contains key files. These are simple JSON
documents which contain all data that is needed to derive the
repository’s master encryption and message authentication keys from a
user’s password. The JSON document from the repository can be
pretty-printed for example by using the Python module json
(shortened to increase readability):

When the repository is opened by restic, the user is prompted for the
repository password. This is then used with scrypt, a key derivation
function (KDF), and the supplied parameters (N, r, p and
salt) to derive 64 key bytes. The first 32 bytes are used as the
encryption key (for AES-256) and the last 32 bytes are used as the
message authentication key (for Poly1305-AES). These last 32 bytes are
divided into a 16 byte AES key k followed by 16 bytes of secret key
r. The key r is then masked for use with Poly1305 (see the paper
for details).

Those keys are used to authenticate and decrypt the bytes contained in
the JSON field data with AES-256 and Poly1305-AES as if they were
any other blob (after removing the Base64 encoding). If the
password is incorrect or the key file has been tampered with, the
computed MAC will not match the last 16 bytes of the data, and restic
exits with an error. Otherwise, the data yields a JSON document
which contains the master encryption and message authentication keys for
this repository (encoded in Base64). The command
resticcatmasterkey can be used as follows to decrypt and
pretty-print the master key:

All data in the repository is encrypted and authenticated with these
master keys. For encryption, the AES-256 algorithm in Counter mode is
used. For message authentication, Poly1305-AES is used as described
above.

A repository can have several different passwords, with a key file for
each. This way, the password can be changed without having to re-encrypt
all data.

A snapshot represents a directory with all files and sub-directories at
a given point in time. For each backup that is made, a new snapshot is
created. A snapshot is a JSON document that is stored in an encrypted
file below the directory snapshots in the repository. The filename
is the storage ID. This string is unique and used within restic to
uniquely identify a snapshot.

The command resticcatsnapshot can be used as follows to decrypt
and pretty-print the contents of a snapshot file:

Here it can be seen that this snapshot represents the contents of the
directory /tmp/testdata. The most important field is tree. When
the meta data (e.g. the tags) of a snapshot change, the snapshot needs
to be re-encrypted and saved. This will change the storage ID, so in
order to relate these seemingly different snapshots, a field
original is introduced which contains the ID of the original
snapshot, e.g. after adding the tag DE to the snapshot above it
becomes:

Once introduced, the original field is not modified when the
snapshot’s meta data is changed again.

All content within a restic repository is referenced according to its
SHA-256 hash. Before saving, each file is split into variable sized
Blobs of data. The SHA-256 hashes of all Blobs are saved in an ordered
list which then represents the content of the file.

In order to relate these plaintext hashes to the actual location within
a Pack file , an index is used. If the index is not available, the
header of all data Blobs can be read.

A tree contains a list of entries (in the field nodes) which contain
meta data like a name and timestamps. When the entry references a
directory, the field subtree contains the plain text ID of another
tree object.

When the command resticcatblob is used, the plaintext ID is needed
to print a tree. The tree referenced above can be dumped as follows:

The restic repository structure is designed in a way that allows
parallel access of multiple instance of restic and even parallel writes.
However, there are some functions that work more efficient or even
require exclusive access of the repository. In order to implement these
functions, restic processes are required to create a lock on the
repository before doing anything.

Locks come in two types: Exclusive and non-exclusive locks. At most one
process can have an exclusive lock on the repository, and during that
time there must not be any other locks (exclusive and non-exclusive).
There may be multiple non-exclusive locks in parallel.

A lock is a file in the subdir locks whose filename is the storage
ID of the contents. It is encrypted and authenticated the same way as
other files in the repository and contains the following JSON structure:

The field exclusive defines the type of lock. When a new lock is to
be created, restic checks all locks in the repository. When a lock is
found, it is tested if the lock is stale, which is the case for locks
with timestamps older than 30 minutes. If the lock was created on the
same machine, even for younger locks it is tested whether the process is
still alive by sending a signal to it. If that fails, restic assumes
that the process is dead and considers the lock to be stale.

When a new lock is to be created and no other conflicting locks are
detected, restic creates a new lock, waits, and checks if other locks
appeared in the repository. Depending on the type of the other locks and
the lock to be created, restic either continues or fails.

For creating a backup, restic scans the source directory for all files,
sub-directories and other entries. The data from each file is split into
variable length Blobs cut at offsets defined by a sliding window of 64
byte. The implementation uses Rabin Fingerprints for implementing this
Content Defined Chunking (CDC). An irreducible polynomial is selected at
random and saved in the file config when a repository is
initialized, so that watermark attacks are much harder.

Files smaller than 512 KiB are not split, Blobs are of 512 KiB to 8 MiB
in size. The implementation aims for 1 MiB Blob size on average.

For modified files, only modified Blobs have to be saved in a subsequent
backup. This even works if bytes are inserted or removed at arbitrary
positions within the file.

The design goals for restic include being able to securely store backups
in a location that is not completely trusted, e.g. a shared system where
others can potentially access the files or (in the case of the system
administrator) even modify or delete them.

General assumptions:

The host system a backup is created on is trusted. This is the most
basic requirement, and essential for creating trustworthy backups.

The restic backup program guarantees the following:

Accessing the unencrypted content of stored files and metadata should
not be possible without a password for the repository. Everything
except the metadata included for informational purposes in the key
files is encrypted and authenticated.

Modifications (intentional or unintentional) can be detected
automatically on several layers:

For all accesses of data stored in the repository it is checked
whether the cryptographic hash of the contents matches the storage
ID (the file’s name). This way, modifications (bad RAM, broken
harddisk) can be detected easily.

Before decrypting any data, the MAC on the encrypted data is
checked. If there has been a modification, the MAC check will
fail. This step happens even before the data is decrypted, so data
that has been tampered with is not decrypted at all.

However, the restic backup program is not designed to protect against
attackers deleting files at the storage location. There is nothing that
can be done about this. If this needs to be guaranteed, get a secure
location without any access from third parties. If you assume that
attackers have write access to your files at the storage location,
attackers are able to figure out (e.g. based on the timestamps of the
stored files) which files belong to what snapshot. When only these files
are deleted, the particular snapshot vanished and all snapshots
depending on data that has been added in the snapshot cannot be restored
completely. Restic is not designed to detect this attack.

Restic can interact with HTTP Backend that respects the following REST
API. The following values are valid for {type}: data, keys,
locks, snapshots, index, config. {path} is a path to
the repository, so that multiple different repositories can be accessed.
The default path is /.

Deletes the repository on the server side. The server responds with “200
OK” if the repository was successfully removed. If this function is not
implemented the server returns “501 Not Implemented”, if this it is
denied by the server it returns “403 Forbidden”.