Once again I’m about to dip into and scratch the surface of Cryptography. So here’s the disclamer: This is not my job. I don’t do this for a living. Don’t ever make up your own encryption algorithms. Try not to write your own cryptographic code. Don’t take anything that I say as the advice of a security expert or legal advice or assume that any code in this post or any linked posts are correct in any way shape or form. If you have a project that requires cryptographic security, I suggest you find someone who has been doing this far longer than I have to write the code for you. Then have several 3rd party security firms review or write your security code. In short, I’m not responsible for your mistakes, or the correctness of the code or writing presented here.

As even more of a taste of just how hard it is, I admit that I thought the last article I wrote covered most of the basics of modern encryption in .NET pretty well. The result? Nope. Missed something big. See, cryptography is hard not because it’s hard to say “take a password plus this data and make it so someone can’t see the data without the password”, but because of all the ways someone could get access to your password, or access your data when its not encrypted, or access your computer when it’s unlocked, or crack your password if you pick a bad password to begin with. Take a look at this reddit thread article that nicely rips apart my previous article. In all seriousness, I really appreciate the feedback, especially when it comes to security because it’s hard, and the devil is in the details.

I forgot to cover data integrity in my last article. Encryption is not enough. Redditors referred to it as authentication, which is one of the uses of HMAC’s, but authenticity is not whats important in the example case I presented for my original article. We are interested in “How to detect that someone or something accidentally or maliciously changed the encrypted data”, essentially, we’re interested in cryptographic data integrity. In addition to that there were a few other things wrong with the article that the redditors pointed out, such as my salt values being overly large, just picking CBC without covering other chaining methods, not discussing how to remove keys from memory, and referencing Jeff Atwood. The long and the short of it, they’re completely right. Getting crypto correct is hard, and good crypto systems are worth millions of dollars. You’re probably not getting paid that much to write a little encryption. So use a library thats already been written and provides high level abstractions and don’t write crypto code yourself.

Alright. HMAC, what is it? The MAC stands for a Message Authentication Code, the H stands for Hash. Put it all together and you have a Hash-based Message Authentication Code. It’s a hashing function thats deliberately designed to resist malicious tampering. The key is preventing malicious tampering. A normal hash function (e.g. MD5 or SHA1) would detect accidental byte tweaks, but somebody maliciously tampering with your data could tinker with it, create a new hash code for the data, replace the old hash code, and you would never be the wiser. For this reason, Message Authentication Codes generally fall into the category of keyed hash algorithms since they use a key or derived key and mix it with a hashing or encryption function to produce a value that an attacker can’t reproduce, providing both data integrity, and (if you happen to be in a sender and receiver role where both parties share a key) authentication. There are however, a couple of things to take into account. First, we shouldn’t use the same key twice to encrypt and hash the data. The more times the same key is used, especially against a known piece of data, the more likely an attack can be developed and used to figure out our key. Arguably, the final key used for encryption and the final key used for the message authentication should be different, as different as possible. The best way to do this would be to append or change the key such that both the encryption key and authentication key run through the KDF (Key Derivation Function) from different starting points. As an example, consider the following:

// Derive the passkey from a hash of the passwordBytes plus salt with the number of hashing rounds.
var deriveKey = new Rfc2898DeriveBytes(password, passwordSalt, 10000);
var deriveHMAC = new Rfc2898DeriveBytes(password, hmacSalt, 10000);

Because the hash function mixes the password with the salt, and because we have different salts, after only one round the derived keys will already be different on account of the salt value. So we have a derived key. One for the actual encryption, one to prevent tampering and provide data integrity.

On a side note, it’s arguably better to authenticate the encrypted output (encrypt then authenticate) rather than authenticate the plaintext, then encrypt, or authenticate and encrypt. Again, I know it’s beating a dead horse, but cryptography is hard, and ultimately, the security of a system is going to depend on the security of the entire system, not just the individual parts. So. We have an HMAC key, we have an encryption key, and we know that we want to encrypt, then authenticate the encrypted output. In addition, we want to make sure anything else that could easily be tampered with is also authenticated, such as our Initialization Vector, since any change to it can easily affect our decrypted output. Another small advantage thats almost not worth mentioning is that by authenticating the encrypted output instead of the plain text is that we can detect if anything has changed even before we start decrypting the text. So here’s an example. In this case, I chose to use HMACSHA1, there’s others on the MSDN but I chose this particular one since it uses the same hash algorithm used internally by the KDF I used in my previous post, Rfc2898DeriveBytes, aka (PKDF2).

In this case, we’re using our derived hmacKey, and we’re computing the hash of both the initialization vector concated with our encrypted ciphertext. That gives us everything we need to have a self validating “package” of data that is secured and can’t be tampered without us knowing unless the attacker knows our key or can break AES256 encryption, but at that point this whole discussion is pointless.

With decryption, remember how I said we compute the Encryption and HMAC key separately? If we did that, and if we computed the hmac over the encrypted data, we can perform the validation step on the data before we compute our decryption key. The only reason we would do this is so that if the data is invalid or has been tampered with we don’t take the time to also compute the decryption key. Small things, but I wanted to explain why the key computation for the encryption and hmac is kept separate:

So, there you go. Basic explanation about why HMAC’s are important, what I missed, some code, and the disclaimer to write security code at your own risk. Full demo code, demo code output, and a bunch of random links after the break.

First, before I begin anything, I want to point out that cryptography is hard. REALLY hard. There are so many possible points of failure, and those points of failure and methods of attack can change depending on what the purpose of encrypting the data is. If you do it wrong, you get lawsuits, and/or you end up on the front pages of major news organizations for data security breaches. Or a rival nation gets your top-secret plans to rule the world. You get the idea. I’ll touch on some of the technical points of failure briefly; I’m mainly going to be exploring this topic through a fairly simple scenario: A password vault file, encrypted with a password, and stored locally on my computer (potentially synced to other computers through some syncing service).

As with any good crypto, the first line of defense is always preventing an attacker from gaining access to the encrypted data in the first place. If data is going over a network, encryption should be part of that as well, and that brings in a whole new host of issues that are outside of the scope of this article. Since I’m planning on storing the data locally on my hard drive, the first line of defense is me and my computer: Lock my computer when I walk away. Make sure there’s no spyware or malware on my computer, no keyloggers or other devices that could steal my password, etc… Going back to my original point about crypto being hard, it’s hard not only because writing the code is hard, it’s difficult because of all the ancillary ways someone could potentially get access to my data that I have to account for.

If I leave my computer unlocked, someone could steal data off of it. Install a keylogger. If I leave this password app open someone could just go in and look at my passwords after I’ve decrypted them. If they have physical access someone could install a HARDWARE keylogger between my keyboard and computer. Pull out my hard drive and clone it. Hack into the company Wi-Fi network and remote access my computer. Or even STEAL my computer. Or TSA could confiscate it. And/or force / legally compel me to reveal my password. You get the idea. There is no such thing as absolute security or secrecy.

However, let’s assume for a moment that an attacker DOES somehow manage to get access to my vault file. (Maybe by getting access to my computer when the vault program is closed and the data encrypted, or a folder syncing program has a hiccup and spams a public link to it on the Internet) My password file is in the hands of someone actively trying to get at my data, and the only thing protecting the data is my password and good cryptography. So. How do I use good cryptographic practices to secure my data?

Since we want to use the latest and greatest, we pick AES (Advanced Encryption Standard, successor to Date Encryption Standard and Triple Data Encryption Standard or Triple DES), Rijndael 256 to be specific (It’s good enough for government work…). In the System.Security.Cryptography namespace, Microsoft has kindly supplied us with the RijndaelManaged class and a pre-provided implementation of the AES standard. I ALWAYS recommend using a pre provided class instead of attempting to roll your own. The ones from Microsoft have gone through EXTENSIVE testing, code, and security review to verify the correctness of the algorithms and code. Frankly, if you write your own, yours will probably have bugs. Bad ones. Bugs that might get you put on the front page of major news organizations for data security breaches if you do it wrong. Warning given, point made, moving on.

Now, there are a couple of things you’ll need in order to do symmetric encryption when we crack open the RijndaelManaged class and try to create an encryption transform:

A Key

An Initialization Vector or IV

Your Data

The Key

First, the key, which is represented as a byte array of some fixed size. The size of your key is important, the number of bits has to match what the algorithm supports, in the case of Rijndael, it supports key sizes of 128, 192, and 256. In our case, we’ll be using a key size of 256. The number of bytes we need is 256 divided by 8 (remember, the key size for the algorithm is specified in bits, but our code mainly works with bytes). So if I’m declaring a byte array for the key for AES 256, it would be new byte[32].

Derive your Key

Now wait, how the heck to I remember a 32 number combination? I can’t. You probably can’t, and unless you’re a brainiac with an eidetic memory I wouldn’t suggest trying with any reasonable hope of success. So, since I have now come to this sad conclusion about my memory, I need some way to take a password or key that I can memorize and turn it into a 32 byte key. Fortunately, there’s a way to do just that by way of a password based key derivation function called PBKDF2. It’s also a standard (RFC 2898, which is only important because the code uses the name of the spec for the class name and not PBKDF2 like I would expect… go figure). Essentially, PBKDF2 is a piece of code that hashes an arbitrary length of text into a pseudo random key. Now, we could use something like MD5 to do something similar, but PBKDF2is designed with cryptography in mind, and to be slow on purpose. Why does it need to be slow? Simply this, cryptography is a big huge lever that makes it easy to go one way, and hard to go another. MD5 is FAST to compute, and someone trying to break into your encrypted data wants to be able to test as many passwords as possible as fast as possible. PBKDF2is slow and more difficult to compute, and the harder it is, the longer it takes to generate a hash, which dramatically reduces the number of passwords a hacker trying to attack your vault can try per second. (There’s a really good article by Jeff Atwood here: http://www.codinghorror.com/blog/2012/04/speed-hashing.html on hashing algorithms in security, rainbow tables, speed hashing, why it’s important to have a slow hash for security, and so on. If you’re interested on reading more about it)

How do we create this hashed key with PBKDF2? The constructor takes 3 things, a string password, an iteration count, and a byte array for something called a salt (Not in that order). The Password is easy, it’s a string or byte array and it’s whatever you choose to use as your “uB3rAw3$omeP@assw0rd!“. iteration count is pretty self-explanatory too, it defaults to 1000, and the bigger the number, the more calculation rounds it does and the longer it takes to calculate the hash. But what about this salt thing? What is it, and why is it important? Do I even care? So here we go.

Salt your Vault

If you’ve heard about salt or salting passwords before, it was probably in reference to a website or service, usually some piece of code that you controlled that sat between your users (or a supposed attacker) and the hashed passwords in your database so that an attacker couldn’t just *make up* a password like zn3lzl that would hash to the same value as the users password like mycoolpassword2 and allow them to log in. If we assume that the attacker has access to your password vault, we can probably assume he has access to your program as well, and it won’t be very hard to figure out some static salt value you’ve stored in your code. So it’s useless right? In some ways yes, in some ways no. IF it’s JUST you, or the attacker is targeting JUST YOUR vault, then yes, it doesn’t matter. However, if for some reason your password vault app becomes popular, and an attacker gains access to, lets say 5000 different peoples vault files, it’s much much easier to check the same passwords across ALL the vaults at once if the salt values are the same. That is again assuming he is not targeting just one particular vault. If they’re all different, the time required to check a password across all the vaults goes up by a factor of 5000, making it less viable to quickly crack multiple vaults. My thought is to simply store the salt along with the vault, since if an attacker has access to your vault, he probably has access to wherever I’d store the salt anyways, and it at least requires him to compute a separate hash for each vault. So, I’ll generate and store a separate, cryptographically random generated salt with my vault file.

As an aside, realize that no amount of encryption can save your data from a bad password. Good hackers are GOING to use huge word lists and fantastic substitution / transposition / combination rules before they even attempt to use a brute force attack. ‘MyL1ttl3P0ny‘ is not a good password. ‘abc123‘ is arguably much worse; but then you’re probably not reading this if abc123 is something you’d use for a password.

Now that we have our password, the salt (loaded from our vault file) and the number of iterations, we can derive a key with the correct size:

Tada! magic. We now have a byte key from the password that we supplied earlier. Ok, now that we have our key, we can see that when you try to create our encryption or decryption transform that it is requiring something called an initialization vector (IV). I know when I was going through all this that I was thinking “what the heck is an initialization vector?” Since I had to teach myself what it was and why it was important, I’m going to assume somebody also doesn’t know and brain dump what I’ve learned on you all as well.

Block Ciphers, Chaining, and Initialization Vectors

First, we have to understand a few things, how a block ciphers works, how cipher block chaining works, how the IV plays in, and why it’s all important. To begin with, block ciphers. Most computer ciphers these days are done using cipher blocks of a given bit size, usually in round multiples of 2, for example 128 bit or 16 bytes chunks (this is actually the size AES uses). Our clear text gets chunked up into these small, manageable blocks of data. Then encryption transform is then run multiple times on each block, and the resulting blocks are all concatenated together to form the final encrypted output.

Because encrypted blocks are computed independently, changes somewhere in the original unencrypted data might only change the encrypted data for a few blocks of the resulting encrypted output. Furthermore, if you happened to have the same 16 byte chunk repeated again such that it aligned with a different blocks, you will get the exact same block of encrypted output. On one hand, this could be useful if you want to send delta updates to an encrypted file, it also means that attacks against individual blocks are feasibly be easier or useful information could be obtained by comparing one version of the vault to another or even patterns might still exist even within the encrypted date. A really awesome example of why it’s important to apply some additional steps during the encryption process with block ciphers are these three images below: (Courtesy of Wikipedia):

Original | No Block Chaining | Proper Cipher Block Chaining

This is where CBC or cipher block chaining comes in. CBC fixes this problem by taking the previously encrypted block and performing a bitwise XOR between it and the raw bytes of the plain text of the next block to be encrypted. Effectively, this means that a one byte change in the first block will change EVERY OTHER block in the resulting ciphertext. However, there’s one last piece. If the first couple of blocks are NOT changed in any way, you can still leek some information about changes that were made by way of the first changed block. So if I have blocks ABCDE and block C gets changed, CBC will cause the resulting ciphertext for blocks D and E to change. It will NOT affect blocks A and B. So, at last, we finally get to the purpose of the initialization vector. The IV is essentially a completely random block to start off the cipher chain. Makes sense now doesn’t it? If I have a completely random block that I create and pass along, it guarantees that even if the plain text doesn’t change at all, the ciphertext will be completely different each time, assuming I generate a new IV each time I change the cipher text. In practice, encrypted data should be statistically indistinguishable from random noise.

Illustration of the process of a block cipher with CBC for Encryption:

Illustration of the process of a block cipher with CBC for Decryption:

Now that we know why we need an initialization vector, we also know, or can guess, what size it should be without having to look at the documentation (16 bytes, because that’s the block size for AES, and our IV is essentially a known random starting block). Since we should generate a new block and also store it with our vault, I’m going to declare our IV and initialize it the same way we initialized our salt:

The CryptoStream

Now that we have all the pieces we can put it all together and encrypt our data to a MemoryStream (the memory stream could be anything, including a FileStream, but this is easier for demo and console output):

EDIT: Thanks to Nick (see comments), he pointed out that cryptoStream.FlushFinalBlock() is unnecessary if you dispose the cryptoStream before you read the data out from the underlying stream. (This happens automatically with a using statement)

Padding under the covers

Some final notes block cipher padding: You’ll notice that when I’m reading the stream I have the following line of code: decryptedBytes.Take(length).ToArray() When you’re using a block cipher like AES, just like the key has to be exactly a certain size, each block of initial data ALSO has to be a certain size. This means you ARE going to get some extra data tacked on the end of your stream that you weren’t anticipating. There’s a couple of ways this can be handled, RijndaelManaged has a couple of padding modes that it can use ranging from filling all the additional slots of data with zeros, or filling them with completely random data, but by default it uses PaddingMode.PKCS7 which fills each extra bytes needed to make the length of the data an even multiple of 16 bytes with the number that represents the number of padded bytes added to fill the empty space. If you have the exact amount of data to exactly to fill the number of blocks, the padding algorithm will add an extra block to ensure that the last byte in the last block that is read represents the amount of padding. Otherwise, depending on whatever data you’re storing, you could accidentally lose some of your data if it was misinterpreted as padding. The crypto stream is aware of the padding and will return the correct length of the original data on the last read call. I simplified the stream reading process for simplicity of demonstrating the use of the crypto stream and how it handles padding, it just reads everything and trims the result with decryptedBytes.Take(length).ToArray(). In real life, you should use or make a ‘real’ stream reader that reads data out of the stream in chunks and aggregates them together, or just serialize / deserialize your objects directly to and from the crypto stream.