If I have received an AES encrypted message, and if I do know the key with which it was encrypted, do I also need to know the mode of operation with which it was encrypted in order to decrypt it? My understanding is that this is indeed the case.

If so, how is the mode of operation transmitted along with the encrypted file?

2 Answers
2

Yes, of course you need to know the mode of operation in order to decrypt.

On the other hand, the mode of operation isn't usually explicitly transmitted in the ciphertext. Using the same key with different modes of operation may cause unexpected weaknesses. For example, consider one message encrypted with CFB mode with IV=2 (which is perfectly secure) and another message encrypted with CTR mode with IV=0 (also secure). If we give the attacker both ciphertexts, he knows a relationship between the first block of the CFB mode plaintext and the third block of the CTR mode plaintext; this is a leakage that does not happen with either mode individually.

Because of this, we restrict a specific key to one mode; that is, both sides agree that this specific key will be used with that specific mode. If there are multiple modes possible, the two sides will agree to the mode at that time.

Now, there are some cases where a key is used only once; one such case happens we're using public keys to encrypt the file. In this case, the encryptor will pick a random AES key, public key encrypt that AES key (using the receiver's public key), and then use the random AES key to encrypt the actual message. In that case, you can encrypt the mode (and other details, such as whether we're using AES and not 3DES) along with the random key.

Would encrypting the mode and algorithm add any significant security in the context you mentioned? Or is it simply a matter of convenience to store the mode/algo with the encrypted key (which must be encrypted)? I had understood that encrypting non-sensitive data (such as an IV, mode, etc) is generally discouraged.
–
hunterApr 7 '13 at 16:46

@hunter: encrypting the mode and the algorithm can often be guessed by an attacker (e.g. "when Alice encrypts a message, she uses AES-128 in CBC mode"), and hence keeping it secret usually doesn't add that much security. As for 'discouraging' encrypting non-sensitive data, well, that's more in the lines of 'don't make things more complex than you have to; complexity is the enemy of security' rather than 'encrypting this will actively make things less secure'.
–
ponchoApr 7 '13 at 17:28

Most systems have a predefined mode of operation, in which case transmitting the mode with the ciphertext isn't necessary. However, if your system does allow for different modes and/or algorithms, you can simply append/prepend the metadata to the ciphertext with a delimiter that you're sure won't appear in the encoded ciphertext.

It would be wise, however, to include this metadata (IV, mode, algo, etc) when calculating the HMAC of your ciphertext, or include it in the header if you're using an authenticated mode such as GCM. Otherwise, if you don't do this, the mode and/or algorithm could be tampered with, resulting in incorrect decryption (or worse).

How do you quantify "most systems"? I would guess most encrypted traffic is transmitted in systems that at least have version dependent algorithms and modes, but perhaps you use a different measure.
–
Henrick HellströmApr 7 '13 at 7:12

I would quantify 'most systems' as there being more systems (at a guess) that use pre-defined protocols (version-dependent or otherwise) than those that don't.
–
hunterApr 7 '13 at 16:32

Per software, per server deployment or per MB of total Internet traffic?
–
Henrick HellströmApr 7 '13 at 16:50