Stashed session keys

Stashed session keys

Now that cleartext indexing is merged, let's add the ability to stash
session keys!

Background
==========

Encrypted e-mail messages are "hybrid" encryption. The message body
is encrypted with an ephemeral session key, and then that session key
is itself encrypted to the user's public key.

If an MUA retains (or obtains) a copy of the session key for a given
message, it can access the cleartext of that message without needing
any access to the user's private key material.

This offers possible wins in efficiency, usability, convenience *and*
security, as the series hopefully makes clear.

Decryption Policies
===================

At the end of the series, there are four sensible policies defined for
message decryption and stashing of session keys. There are only two i
expect to see any widespread regular use: "auto", and "true". But
hopefully the reasons for including the other two policies ("false"
and "nostash") are made clear by the series itself.

I'll replicate here the table this series adds to notmuch-config(1),
in describing the available values for index.try_decrypt:

[PATCH 01/18] mime-node: handle decrypt_result more safely

If (for whatever reason) we don't get a decrypt_result back, or it's
not structured the way we expect it to be, we shouldn't choke on it.
---
mime-node.c | 14 ++++++++------
1 file changed, 8 insertions(+), 6 deletions(-)

[PATCH 03/18] crypto: use stashed session-key properties for decryption, if available

When doing any decryption, if the notmuch database knows of any
session keys associated with the message in question, try them before
defaulting to using default symmetric crypto.

The primary advantage this offers is a significant speedup when
rendering large encrypted threads ("notmuch show") if session keys
happen to be cached.

Additionally, it permits message composition without access to
asymmetric secret keys ("notmuch reply"); and it permits recovering a
cleartext index when reindexing after a "notmuch restore" for those
messages that already have a session key stored.

This does nothing if no session keys are stashed in the database,
which is fine. Actually stashing session keys in the database will
come as a subsequent patch.

Note that this only works for GMime 2.6.21 and later (the session key
interface wasn't available before then). I don't think we're ready
for this to be a minimum version requirement yet, so instead if you
build against a prior version of GMime, it simply won't try to use
stashed session keys.
---
doc/man7/notmuch-properties.rst | 31 +++++++++++++++++++++++++++++++
lib/index.cc | 2 +-
mime-node.c | 13 ++++++++++---
util/crypto.c | 31 ++++++++++++++++++++++++++++++-
util/crypto.h | 3 ++-
5 files changed, 74 insertions(+), 6 deletions(-)

diff --git a/doc/man7/notmuch-properties.rst b/doc/man7/notmuch-properties.rst
index 68121359..31d7a104 100644
--- a/doc/man7/notmuch-properties.rst
+++ b/doc/man7/notmuch-properties.rst
@@ -74,6 +74,35 @@ of its normal activity.
**notmuch-config(1)**), then this property will not be set on that
message.

+**session-key**
+
+ When **notmuch-show(1)** or **nomtuch-reply** encounters a message
+ with an encrypted part and ``--decrypt`` is set, if notmuch finds a
+ ``session-key=`` property associated with the message, it will try
+ that stashed session key for decryption.
+
+ Using a stashed session key with "notmuch show" will speed up
+ rendering of long encrypted threads. It also allows the user to
+ destroy the secret part of any expired encryption-capable subkey
+ while still being able to read any retained messages for which
+ they have stashed the session key. This enables truly deletable
+ e-mail, since (once the session key and asymmetric subkey are both
+ destroyed) there are no keys left that can be used to decrypt any
+ copy of the original message previously stored by an adversary.
+
+ However, access to the stashed session key for an encrypted message
+ permits full byte-for-byte reconstruction of the cleartext
+ message. This includes attachments, cryptographic signatures, and
+ other material that cannot be reconstructed from the index alone.
+
+ The session key should be in the ASCII text form produced by
+ GnuPG. For OpenPGP, that consists of a decimal representation of
+ the hash algorithm used (identified by number from RFC 4880,
+ e.g. 9 means AES-256) followed by a colon, followed by a
+ hexidecimal representation of the algorithm-specific key. For
+ example, an AES-128 key might be stashed in a notmuch property as:
+ ``session-key=7:14B16AF65536C28AF209828DFE34C9E0``.
+
SEE ALSO
========

+ If a session key is already known for the message, then it
+ will be decrypted automatically unless the user explicitly
+ sets ``--decrypt=false``.
+
Decryption expects a functioning **gpg-agent(1)** to provide any
- needed credentials. Without one, the decryption will fail.
+ needed credentials. Without one, the decryption will likely fail.

[PATCH 09/18] cli/show: use decryption policy "auto" by default.

When showing a message, if the user doesn't specify --decrypt= at all,
but a stashed session key is known to notmuch, notmuch should just go
ahead and try to decrypt the message with the session key (without
bothering the user for access to their asymmetric secret key).

The user can disable this at the command line with --decrypt=false if
they really don't want to look at the e-mail that they've asked
notmuch to show them.

+ If a session key is already known for the message, then it
+ will be decrypted automatically unless the user explicitly
+ sets ``--decrypt=false``.
+
Decryption expects a functioning **gpg-agent(1)** to provide any
needed credentials. Without one, the decryption will fail.

When **notmuch-show(1)** or **nomtuch-reply** encounters a message
- with an encrypted part and ``--decrypt`` is set, if notmuch finds a
- ``session-key=`` property associated with the message, it will try
- that stashed session key for decryption.
+ with an encrypted part, if notmuch finds a ``session-key=``
+ property associated with the message, it will try that stashed
+ session key for decryption.
+
+ If you do not want to use any stashed session keys that might be
+ present, you should pass those programs ``--decrypt=false``.

Using a stashed session key with "notmuch show" will speed up
rendering of long encrypted threads. It also allows the user to
--
2.14.2

- If true and the message is encrypted, try to decrypt the
- message while indexing. If decryption is successful, index
+ If ``true`` and the message is encrypted, try to decrypt the
+ message while indexing. If ``auto``, and notmuch already
+ knows about a session key for the message, it will try
+ decrypting using that session key but will not try to access
+ the user's secret keys. If decryption is successful, index
the cleartext itself. Either way, the message is always
- stored to disk in its original form (ciphertext). Be aware
- that the index is likely sufficient to reconstruct the
- cleartext of the message itself, so please ensure that the
+ stored to disk in its original form (ciphertext).
+
+ Be aware that the index is likely sufficient to reconstruct
+ the cleartext of the message itself, so please ensure that the
notmuch message index is adequately protected. DO NOT USE
``--try-decrypt=true`` without considering the security of
your index.
diff --git a/doc/man1/notmuch-new.rst b/doc/man1/notmuch-new.rst
index bc26aa48..d8cb77f5 100644
--- a/doc/man1/notmuch-new.rst
+++ b/doc/man1/notmuch-new.rst
@@ -43,11 +43,15 @@ Supported options for **new** include
``--quiet``
Do not print progress or results.

- If true, when encountering an encrypted message, try to
+ If ``true``, when encountering an encrypted message, try to
decrypt it while indexing. If decryption is successful, index
- the cleartext itself. Be aware that the index is likely
+ the cleartext itself. If ``auto``, try to use any session key
+ already known to belong to this message, but do not attempt to
+ use the user's secret keys.
+
+ Be aware that the index is likely
sufficient to reconstruct the cleartext of the message itself,
so please ensure that the notmuch message index is adequately
protected. DO NOT USE ``--try-decrypt=true`` without
diff --git a/doc/man1/notmuch-reindex.rst b/doc/man1/notmuch-reindex.rst
index 21f6c7a9..b15981a2 100644
--- a/doc/man1/notmuch-reindex.rst
+++ b/doc/man1/notmuch-reindex.rst
@@ -21,15 +21,20 @@ messages using the supplied options.

Supported options for **reindex** include

- ``--try-decrypt=(true|false)``
-
- If true, when encountering an encrypted message, try to
- decrypt it while reindexing. If decryption is successful,
- index the cleartext itself. Be aware that the index is likely
- sufficient to reconstruct the cleartext of the message itself,
- so please ensure that the notmuch message index is adequately
- protected. DO NOT USE ``--try-decrypt=true`` without
- considering the security of your index.
+ ``--try-decrypt=(true|auto|false)``
+
+ If ``true``, when encountering an encrypted message, try to
+ decrypt it while reindexing. If ``auto``, and notmuch already
+ knows about a session key for the message, it will try
+ decrypting using that session key but will not try to access
+ the user's secret keys. If decryption is successful, index
+ the cleartext itself.
+
+ Be aware that the index is likely sufficient to reconstruct
+ the cleartext of the message itself, so please ensure that the
+ notmuch message index is adequately protected. DO NOT USE
+ ``--try-decrypt=true`` without considering the security of
+ your index.

The new "auto" decryption policy is not only good for "notmuch show"
and "notmuch reindex". It's also useful for indexing messages --
there's no good reason to not try to go ahead and index the cleartext
of a message that we have a stashed session key for.

There are some situations where the user wants to get rid of the
cleartext index of a message. For example, if they're indexing
encrypted messages normally, but suddenly they run across a message
that they really don't want any trace of in their index.

But of course, clearing the cleartext index without clearing the
stashed session key is just silly. So we do the expected thing and
also destroy any stashed session keys while we're destroying the index
of the cleartext.

As a workaround, after removing session keys and cleartext material
from the database, the user probably should do something like "notmuch
compact" to try to purge whatever recoverable data is left in the
xapian freelist. This problem really needs to be addressed within
xapian, though, if we want it fixed right.
---
doc/man1/notmuch-reindex.rst | 3 +++
lib/message.cc | 5 +++++
test/T357-index-decryption.sh | 17 +++++++++++++++++
3 files changed, 25 insertions(+)

If you're going to store the cleartext index of an encrypted message,
in most situations you might just as well store the session key.
Doing this storage has efficiency and recoverability advantages.

Combined with a schedule of regular OpenPGP subkey rotation and
destruction, this can also offer security benefits, like "deletable
e-mail", which is the store-and-forward analog to "forward secrecy".

But wait, i hear you saying, i have a special need to store cleartext
indexes but it's really bad for me to store session keys! Maybe
(let's imagine) i get lots of e-mails with incriminating photos
attached, and i want to be able to search for them by the text in the
e-mail, but i don't want someone with access to the index to be
actually able to see the photos themselves.

diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index ae7586f3..5adde070 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -143,10 +143,12 @@ The available configuration items are described below.
**[STORED IN DATABASE]**
When indexing an encrypted e-mail message, if this variable is
set to ``true``, notmuch will try to decrypt the message and
- index the cleartext. If ``auto``, it will try to index the
- cleartext if a stashed session key is already known for the message,
- but will not try to access your secret keys. Use ``false`` to
- avoid decrypting even when a session key is already known.
+ index the cleartext, stashing a copy of any discovered session
+ keys for the message. If ``auto``, it will try to index the
+ cleartext if a stashed session key is already known for the message
+ (e.g. from a previous copy), but will not try to access your
+ secret keys. Use ``false`` to avoid decrypting even when a
+ stashed session key is already present.

If ``true`` and the message is encrypted, try to decrypt the
- message while indexing. If ``auto``, and notmuch already
- knows about a session key for the message, it will try
- decrypting using that session key but will not try to access
- the user's secret keys. If decryption is successful, index
- the cleartext itself. Either way, the message is always
- stored to disk in its original form (ciphertext).
+ message while indexing, storing any session keys discovered.
+ If ``auto``, and notmuch already knows about a session key for
+ the message, it will try decrypting using that session key but
+ will not try to access the user's secret keys. If decryption
+ is successful, index the cleartext itself. Either way, the
+ message is always stored to disk in its original form
+ (ciphertext).

Be aware that the index is likely sufficient to reconstruct
the cleartext of the message itself, so please ensure that the
diff --git a/doc/man1/notmuch-new.rst b/doc/man1/notmuch-new.rst
index d8cb77f5..351b7591 100644
--- a/doc/man1/notmuch-new.rst
+++ b/doc/man1/notmuch-new.rst
@@ -46,16 +46,17 @@ Supported options for **new** include
``--try-decrypt=(true|auto|false)``

If ``true``, when encountering an encrypted message, try to
- decrypt it while indexing. If decryption is successful, index
- the cleartext itself. If ``auto``, try to use any session key
- already known to belong to this message, but do not attempt to
- use the user's secret keys.
-
- Be aware that the index is likely
- sufficient to reconstruct the cleartext of the message itself,
- so please ensure that the notmuch message index is adequately
- protected. DO NOT USE ``--try-decrypt=true`` without
- considering the security of your index.
+ decrypt it while indexing, and store any discovered session
+ keys. If ``auto``, try to use any session key already known
+ to belong to this message, but do not attempt to use the
+ user's secret keys. If decryption is successful, index the
+ cleartext of the message.
+
+ Be aware that the index is likely sufficient to reconstruct
+ the cleartext of the message itself, so please ensure that the
+ notmuch message index is adequately protected. DO NOT USE
+ ``--try-decrypt=true`` without considering the security of
+ your index.

If ``true``, when encountering an encrypted message, try to
- decrypt it while reindexing. If ``auto``, and notmuch already
- knows about a session key for the message, it will try
- decrypting using that session key but will not try to access
- the user's secret keys. If decryption is successful, index
- the cleartext itself.
+ decrypt it while reindexing, storing any session keys
+ discovered. If ``auto``, and notmuch already knows about a
+ session key for the message, it will try decrypting using that
+ session key but will not try to access the user's secret keys.
+ If decryption is successful, index the cleartext itself.

If ``false``, notmuch reindex will also delete any stashed
session keys for all messages matching the search terms.
diff --git a/doc/man7/notmuch-properties.rst b/doc/man7/notmuch-properties.rst
index e2ab43be..5dcbb4ae 100644
--- a/doc/man7/notmuch-properties.rst
+++ b/doc/man7/notmuch-properties.rst
@@ -98,6 +98,10 @@ of its normal activity.
message. This includes attachments, cryptographic signatures, and
other material that cannot be reconstructed from the index alone.

Here's the configuration choice for people who want a cleartext index,
but don't want stashed session keys.

Interestingly, this "nostash" decryption policy is actually the same
policy that should be used by "notmuch show" and "notmuch reply",
since they never modify the index or database when they are invoked
with --decrypt.

**[STORED IN DATABASE]**
+
+ One of ``false``, ``auto``, ``nostash``, or ``true``.
+
When indexing an encrypted e-mail message, if this variable is
set to ``true``, notmuch will try to decrypt the message and
index the cleartext, stashing a copy of any discovered session
@@ -150,11 +153,14 @@ The available configuration items are described below.
secret keys. Use ``false`` to avoid decrypting even when a
stashed session key is already present.

+ ``nostash`` is the same as ``true`` except that it will not
+ stash newly-discovered session keys in the database.
+
Be aware that the notmuch index is likely sufficient to
reconstruct the cleartext of the message itself, so please
ensure that the notmuch message index is adequately protected.
- DO NOT USE ``index.try_decrypt=true`` without considering the
- security of your index.
+ DO NOT USE ``index.try_decrypt=true`` or ``index-only``
+ without considering the security of your index.

If ``true`` and the message is encrypted, try to decrypt the
- message while indexing, storing any session keys discovered.
+ message while indexing, stashing any session keys discovered.
If ``auto``, and notmuch already knows about a session key for
the message, it will try decrypting using that session key but
will not try to access the user's secret keys. If decryption
@@ -61,11 +61,14 @@ Supported options for **insert** include
message is always stored to disk in its original form
(ciphertext).

+ ``nostash`` is the same as ``true`` except that it will not
+ stash newly-discovered session keys in the database.
+
Be aware that the index is likely sufficient to reconstruct
the cleartext of the message itself, so please ensure that the
notmuch message index is adequately protected. DO NOT USE
- ``--try-decrypt=true`` without considering the security of
- your index.
+ ``--try-decrypt=true`` or ``nostash`` without considering
+ the security of your index.

If ``true``, when encountering an encrypted message, try to
- decrypt it while indexing, and store any discovered session
+ decrypt it while indexing, and stash any discovered session
keys. If ``auto``, try to use any session key already known
to belong to this message, but do not attempt to use the
user's secret keys. If decryption is successful, index the
cleartext of the message.

+ ``nostash`` is the same as ``true`` except that it will not
+ stash newly-discovered session keys in the database.
+
Be aware that the index is likely sufficient to reconstruct
the cleartext of the message itself, so please ensure that the
notmuch message index is adequately protected. DO NOT USE
- ``--try-decrypt=true`` without considering the security of
- your index.
+ ``--try-decrypt=true`` or ``nostash`` without considering
+ the security of your index.

If ``true``, when encountering an encrypted message, try to
- decrypt it while reindexing, storing any session keys
+ decrypt it while reindexing, stashing any session keys
discovered. If ``auto``, and notmuch already knows about a
session key for the message, it will try decrypting using that
session key but will not try to access the user's secret keys.
If decryption is successful, index the cleartext itself.

+ ``nostash`` is the same as ``true`` except that it will not
+ stash newly-discovered session keys in the database.
+
If ``false``, notmuch reindex will also delete any stashed
session keys for all messages matching the search terms.

Be aware that the index is likely sufficient to reconstruct
the cleartext of the message itself, so please ensure that the
notmuch message index is adequately protected. DO NOT USE
- ``--try-decrypt=true`` without considering the security of
- your index.
+ ``--try-decrypt=true`` or ``nostash`` without considering
+ the security of your index.

[PATCH 17/18] docs: clean up documentation about decryption policies

Now that the range of sensible decryption policies has come into full
view, we take a bit of space to document the distinctions.

Most people will use either "auto" or "true" -- but we provide "false"
and "nostash" to handle use cases that might reasonably be requested.

Note also that these can be combined in sensible ways. Like, if your
mail comes in regularly to a service that doesn't have access to your
secret keys, but does have access to your index, and you feel
comfortable adding selected encrypted messages to the index after
you've read them, you could stay in "auto" normally, and then when you
find yourself reading an indexable message (e.g. one you want to be
able to search for in the future, and that you don't mind exposing to
whatever entities have access to your inde), you can do:

- One of ``false``, ``auto``, ``nostash``, or ``true``.
+ Policy for decrypting encrypted messages during indexing.
+ Must be one of: ``false``, ``auto``, ``nostash``, or
+ ``true``.

When indexing an encrypted e-mail message, if this variable is
set to ``true``, notmuch will try to decrypt the message and
@@ -156,11 +158,40 @@ The available configuration items are described below.
``nostash`` is the same as ``true`` except that it will not
stash newly-discovered session keys in the database.

:param filename: should be a path relative to the path of the
@@ -427,6 +443,23 @@ class Database(object):
API. You might want to look into the underlying method
:meth:`Message.maildir_flags_to_tags`.

+ :param try_decrypt: If the message contains any encrypted
+ parts, and try_decrypt is set to
+ :attr:`DECRYPTION_POLICY`.TRUE, notmuch will try to
+ decrypt the message and index the cleartext, stashing any
+ discovered session keys. If it is set to
+ :attr:`DECRYPTION_POLICY`.FALSE, it will never try to
+ decrypt during indexing. If it is set to
+ :attr:`DECRYPTION_POLICY`.AUTO, then it will try to use
+ any stashed session keys it knows about, but will not try
+ to access the user's secret keys.
+ :attr:`DECRYPTION_POLICY`.NOSTASH behaves the same as
+ :attr:`DECRYPTION_POLICY`.TRUE except that no session keys
+ are stashed in the database. If try_decrypt is set to
+ None (the default), then the database itself will decide
+ whether to decrypt, based on the `index.try_decrypt`
+ configuration setting (see notmuch-config(1)).
+
:returns: On success, we return

On Wed 2017-10-25 02:52:01 -0400, Daniel Kahn Gillmor wrote:
> - DO NOT USE ``index.try_decrypt=true`` without considering the
> - security of your index.
> + DO NOT USE ``index.try_decrypt=true`` or ``index-only``
> + without considering the security of your index.

jrollins helpfully caught that i'd let the previous, worse name of
"nostash" ("index-only") sneak into this commit. I've fixed it in my
local copy of this branch, which can be found as "session-keys" at
https://gitlab.com/dkg/notmuch (current commit ID
ac7a7bb931b68a17dff0b4b782a2bdeced4e779c)

I'll wait to send revised patches to the list once i've incorporated any
other reviews folks want to send my way.