[Tutorial] Implementing the Advanced Encryption Standard

This is a discussion on [Tutorial] Implementing the Advanced Encryption Standard within the C Programming forums, part of the General Programming Boards category; Note: the latest version of this tutorial can be found at: Advanced Encryption Standard (AES) Tutorial
To kick-start the forum ...

[Tutorial] Implementing the Advanced Encryption Standard

To kick-start the forum being back online again, I'm starting with a tutorial about implementing the de facto standard encryption algorithm, recommended by the National Institute of Standards and Technology, which is called "Advanced Encryption Standard", or commonly referenced as AES. I'll start with a short introduction about cryptography, followed by an explanation of the algorithm, the idea and its creators. Finally, I'll guide you through all the necessary steps to implement this algorithm in C, followed by its integration into a mode of operation for block ciphers to encrypt plaintext with more than 128 bits.

Introduction to cryptography:

Cryptography is the science of information and communication security. (Serge Vaudenay, A classical introduction to cryptography)

Cryptography is the science of secret codes, enabling the confidentiality of communication through an insecure channel. It protects against unauthorized parties by preventing unauthorized alteration of use. Generally speaking, it uses an cryptographic system to transform a plaintext into a ciphertext, using most of the time a key.

One has to notice that there exist certain cipher that don't need a key at all. A famous example is ROT13 (abbreviation from Rotation 13), a simple Caesar-cipher that obscures text by replacing each letter with the letter thirteen places down in the alphabet. Since our alphabet has 26 characters, it is enough to encrypt the ciphertext again to retrieve the original message.

Let me just mention briefly that there are secure public-key ciphers, like the famous and very secure Rivest-Shamir-Adleman (commonly called RSA) that uses a public key to encrypt a message and a secret key to decrypt it.

Cryptography is a very important domain in computer science with many applications. The most famous example of cryptography is certainly the Enigma machine, the legendary cipher machine used by the german Third Reich to encrypt their messages, who's security breach ultimately led to the defeat of their submarine force.

Cryptography has long been of interest to intelligence gathering agencies and law enforcement agencies. Because of its facilitation of privacy, and the diminution of privacy attendant on its prohibition, cryptography is also of considerable interest to civil rights supporters. Accordingly, there has been a history of controversial legal issues surrounding cryptography, especially since the advent of inexpensive computers has made possible widespread access to high quality cryptography.

In some countries, even the domestic use of cryptography is, or has been, restricted. Until 1999, France significantly restricted the use of cryptography domestically. In China, a license is still required to use cryptography. Many countries have tight restrictions on the use of cryptography. Among the more restrictive are laws in Belarus, Kazakhstan, Mongolia, Pakistan, Russia, Singapore, Tunisia, Venezuela, and Vietnam.[31]

In the United States, cryptography is legal for domestic use, but there has been much conflict over legal issues related to cryptography. One particularly important issue has been the export of cryptography and cryptographic software and hardware. Because of the importance of cryptanalysis in World War II and an expectation that cryptography would continue to be important for national security, many western governments have, at some point, strictly regulated export of cryptography. After World War II, it was illegal in the US to sell or distribute encryption technology overseas; in fact, encryption was classified as a munition, like tanks and nuclear weapons.[32] Until the advent of the personal computer and the Internet, this was not especially problematic. Good cryptography is indistinguishable from bad cryptography for nearly all users, and in any case, most of the cryptographic techniques generally available were slow and error prone whether good or bad. However, as the Internet grew and computers became more widely available, high quality encryption techniques became well-known around the globe. As a result, export controls came to be seen to be an impediment to commerce and to research.

Introduction to the Advanced Encryption Standard:

The Advanced Encryption Standard, in the following referenced as AES, is the winner of the contest, held in 1997 by the US Government, after the Data Encryption Standard was found too weak because of its small key size and the technological advancements in processor power. Fifteen candidates were accepted in 1998 and based on public comments the pool was reduced to fived finalists in 1999. In October 2000, one of these five algorithms was selected as the forthcoming standard: a slightly modified version of the Rijndael.

The Rijndael, whose name is based on the names of its two Belgian inventors, Joan Daemen and Vicent Rijmen, is a Block cipher, which means that it works on fixed-length group of bits, which are called blocks. It takes an input block of a certain size, usually 128, and produces a corresponding output block of the same size. The transformation requires a second input, which is the secret key. It is important to know that the secret key can be of any size (depending on the cipher used) and that AES uses three different key sizes: 128, 192 and 256 bits.
To encrypt messages longer than the block size, a mode of operation is chosen, which I will explain at the very end of this tutorial, after the implementation of AES.
While AES supports only block sizes of 128 bits and key sizes of 128, 192 and 256 bits, the original Rijndael supports key and block sizes in any multiple of 32, with a minimum of 128 and a maximum of 256 bits.

Further readings: Unlike DES, which is based on an Feistel network, AES is a substitution-permutation network, which is a series of mathematical operations that use substitutions (also called S-Box) and permutations (P-Boxes) and their careful definition implies that each output bit depends on every input bit.

Description of the Advanced Encryption Standard algorithm

AES is an iterated block cipher with a fixed block size of 128 and a variable key length. The different transformations operate on the intermediate results, called state. The state is a rectangular array of bytes and size the block size is 128 bits, which is 16 bytes, the rectangular array is of size 4x4. (In the Rijndael version with variable block size, the row size is fixed to four and the number of columns vary. The number of columns is the block size divided by 32 and denoted Nb). The cipher key is similarly pictured as a rectangular array with four rows. The number of columns of the cipher key, denoted Nk, is equal to the key length divided by 32.

It is very important to know that the cipher input bytes are mapped onto the the state bytes in the order a0,0, a1,0, a2,0, a3,0, a0,1, a1,1, a2,1, a3,1, a4,1 ... , and the bytes of the cipher key are mapped onto the array in the order k0,0, k1,0, k2,0, k3,0, k0,1, k1,1, k2,1, k3,1, k4,1 ... At the end of the cipher operation, the cipher output is extracted from the state by taking the state bytes in the same order.

AES uses a variable number of rounds, which are fixed:

A key of size 128 has 10 rounds.
A key of size 192 has 12 rounds.
A key of size 256 has 14 rounds.

During each round, the following operations are applied on the state:

1. SubBytes: every byte in the state is replaced by another one, using the Rijndael S-Box
2. ShiftRow: every row in the 4x4 array is shifted a certain amount to the left
3. MixColumn: a linear transformation on the columns of the state
4. AddRoundKey: each byte of the state is combined with a round key, which is a different key for each round and derived from the Rijndael key schedule

The cipher key is expanded into a larger key, which is later used for the actual operations

The roundKey is added to the state before starting the with loop

The FinalRound() is the same as Round(), apart from missing the MixColumns() operation.

During each round, another part of the ExpandedKey is used for the operations

The ExpandedKey shall ALWAYS be derived from the Cipher Key and never be specified directly.

AES operations: SubBytes, ShiftRow, MixColumn and AddRoundKey:

The AddRoundKey operation:
In this operation, a Round Key is applied to the state by a simple bitwise XOR. The Round Key is derived from the Cipher Key by the means of the key schedule. The Round Key length is equal to the block key length (= 16 bytes).

Please note that the inverse of ShiftRow is the same cyclically shift but this time to the right. It will be needed later for decoding.

The SubBytes operation:
The SubBytes operation is a non-linear byte substitution, operating on each byte of the state independently. The substitution table (S-Box) is invertible and is constructed by the composition of two transformations:

Then, applying an affine transformation which is documented in the Rijndael documentation.

Since the S-Box is independent of any input, pre-calculated forms are used, if enough memory (256 bytes for one S-Box) is available.
Each byte of the state is then substituted by the value in the S-Box whose index corresponds to the value in the state:

a(i,j) = SBox[a(i,j)]

Please note that the inverse of SubBytes is the same operation, using the inversed S-Box, which is also precalculated.

The MixColumn operation:
I will keep this section very short since it involves a lot of very advance mathematical calculations in the Rijndael's finite field.
All you have to know is that it corresponds to the matrix multiplication with:

2 3 1 1
1 2 3 1
1 1 2 3
3 1 1 2

and that the addition and multiplication operations are a little different from the normal ones.
You can skip this part if you are not interested in the math involved:

Addition and Substraction:

Addition and subtraction are performed by the exclusive or operation. The two operations are the same; there is no difference between addition and subtraction.

Code:

Multiplication in Rijndael's galois field is a little more complicated. The procedure is as follows:
* Take two eight-bit numbers, a and b, and an eight-bit product p
* Set the product to zero.
* Make a copy of a and b, which we will simply call a and b in the rest of this algorithm
* Run the following loop eight times:
1. If the low bit of b is set, exclusive or the product p by the value of a
2. Keep track of whether the high (eighth from left) bit of a is set to one
3. Rotate a one bit to the left, discarding the high bit, and making the low bit have a value of zero
4. If a's hi bit had a value of one prior to this rotation, exclusive or a with the hexadecimal number 0x1b
5. Rotate b one bit to the right, discarding the low bit, and making the high (eighth from left) bit have a value of zero.
* The product p now has the product of a and b

The Rijndael Key Schedule:

The Key Schedule is responsible for expanding a short key into a larger key, whose parts are used during the different iterations. Each key size is expanded to a different size:

An 128 bit key is expanded to an 176 byte key.
An 192 bit key is expanded to an 208 byte key.
An 256 bit key is expanded to an 240 byte key.

There is a relation between the cipher key size, the number of rounds and the ExpandedKey size. For an 128-bit key, there is one initial AddRoundKey operation plus there are 10 rounds and each rounds need a new 16 byte key, therefor we require 10+1 RoundKeys of 16 byte, which equals 176 byte. The same logic can be applied to the two other cipher key sizes. The general formula is that: ExpandedKeySize = (nbrRounds+1) * BlockSize

The Key Schedule is made up of iterations of the Key schedule core, which works on 4-byte words. The core uses a certain number of operations, which are explained here:

Rcon:
This section is again extremely mathematical and I recommend everyone who is interested to read this description. Just note that the Rcon values can be pre-calculated, which results in a simple substitution (a table lookup) in a pre-calculated, fixed Rcon table (again, Rcon can also be calculated on-the-fly if memory is a design restriction.)

S-Box:
The Key Schedule uses the same S-Box substitution as the main algorithm body.

Now that we know what the operations are, let me show you the key schedule core (in pseudo-C):

Nk is the number of columns in the cipher key (128-bit -> 4, 192-bit -> 5, 256-bit -> 6)

W is of type "word", which is 4-bytes

Let me try to explain this in an easier understandable way:

The first n bytes of the expanded key are simply the cipher key (n = the size of the encryption key)

The rcon value i is set to 1

Until we have enough bytes of expanded key, we do the following to generate n more bytes of expanded key (please note once again that "n" is used here, this varies depending on the key size)

we do the following to generate four bytes

we use a temporary 4-byte word called t

we assign the previous 4 bytes to t

we perform the key schedule core on t, with i as rcon value

we increment i

we XOR t with the 4-byte word n bytes before in the expandedKey (where n is once either either 16,24 or 32 bytes)

we do the following x times to generate the next x*4 bytes of the expandedKey (x = 3 for n=16,32 and x = 5 for n=24)

we assign the previous 4-byte word to t

we XOR t with the 4-byte word n bytes before in the expandedKey (where n is once either either 16,24 or 32 bytes)

if n = 32 (and ONLY then), we do the following to generate 4 more bytes

we assign the previous 4-byte word to t

We run each of the four bytes in t through Rijndael's S-box

we XOR t with the 4-byte word 32 bytes before in the expandedKey

if n = 32 (and ONLY then), we do the following three times to generate twelve more bytes

we assign the previous 4-byte word to t

we XOR t with the 4-byte word 32 bytes before in the expandedKey

We now have our expandedKey

Don't worry if you still have problems understanding the Key Schedule, you'll see that the implementation isn't very hard. What you should note is that:

the part in red is only for cipher key size = 32

for n=16, we generate: 4 + 3*4 bytes = 16 bytes per iteration

for n=24, we generate: 4 + 5*4 bytes = 24 bytes per iteration

for n=32, we generate: 4 + 3*4 + 4 + 3*4 = 32 bytes per iteration

The implementation of the key schedule is pretty straight forward, but since there is a lot of code repetition, it is possible to optimize the loop slightly and use the modulo operator to check when the additional operations have to be made.

Implementation: The Key Schedule

We will start the implementation of AES with the Cipher Key expansion. As you can read in the theoretical part above, we intend to enlarge our input cipher key, whose size varies between 128 and 256 bits into a larger key, from which different RoundKeys can be derived.
I prefer to implement the helper functions (such as rotate, Rcon or S-Box first), test them and then move on to the larger loops. If you are not a fan of bottom-up approaches, feel free to start a little further in this tutorial and move your way up, but I felt that my approach was the more logical one here.

Implementation: General comments

Even though some might think that integers were the best choice to work with, since their 32 bit size best corresponds one word, I strongly discourage you from using integers. You wrongly assume that integers, or more specifically the "int" type, always has 4 bytes. However, the required ranges for signed and unsigned int are identical to those for signed and unsigned short. On compilers for 8 and 16 bit processors (including Intel x86 processors executing in 16 bit mode, such as under MS-DOS), an int is usually 16 bits and has exactly the same representation as a short. On compilers for 32 bit and larger processors (including Intel x86 processors executing in 32 bit mode, such as Win32 or Linux) an int is usually 32 bits long and has exactly the same representation as a long.

For this very reason, we will be using unsigned chars, since the size of an char (which is called CHAR_BIT and defined in limits.h) is required to be at least 8. Jack Klein wrote: Almost all modern computers today use 8 bit bytes (technically called octets, but there are still some in production and use with other sizes, such as 9 bits. Also some processors (especially Digital Signal Processors) cannot efficiently access memory in smaller pieces than the processor's word size. There is at least one DSP I have worked with where CHAR_BIT is 32. The char types, short, int and long are all 32 bits.

Since we want to keep our code as portable as possible and since it is up to the compiler to decide if the default type for char is signed or not, we will specify unsigned char throughout the entire code.

Implementation: S-Box

The S-Box values can either be calculated on-the-fly to save memory or the pre-calculated values can be stored in an array. Since I assume that every machine my code runs on will have at least 2x 256bytes (there are 2 S-Boxes, one for the encryption and one for the decryption) we will store the values in an array. Additionally, instead of accessing the values immediately from our program, I'll wrap a little function around which makes for a more readable code and would allow us to add additional code later on. Of course, this is a matter of taste, feel free to access the array immediately.

Here's the code for the 2 S-Boxes, it's only a table-lookup that returns the value in the array whose index is specified as a parameter of the function:

From the theoretical part, you should know already that Rotate takes a word (a 4-byte array) and rotates it 8 bit to the left. Since 8 bit correspond to one byte and our array type is character (whose size is one byte), rotating 8 bit to the left corresponds to shifting cyclically the array values one to the left.

Same as with the S-Box, the Rcon values can be calculated on-the-fly but once again I decide to store them in an array since they only require 255 bytes of space. To keep in line with the S-Box implementation, I write a little access function.

The implementation of the Key Schedule Core from the pseudo-C is pretty easy. All the code does is apply the operations one after the other on the 4-byte word. The parameters are the 4-byte word and the iteration counter, on which Rcon depends.

The Key Expansion is where it all comes together. As you can see in the pretty big list in the theory about the Rijndael Key Expansion, we need to apply several operations a number of times, depending on they key size.
As the key size can only take a very limited number of values, I decided to implement it as an enumeration type. Not only does that limit the key size to only three possible values, it also makes the code more readable.

Code:

enum keySize{
SIZE_16 = 16,
SIZE_24 = 24,
SIZE_32 = 32
};

Our key expansion function basically needs only two things:

the input cipher key

the output expanded key

Since in C, it is not possible to know the size of an array passed as pointer to a function, we'll add the cipher key size (of type "enum keySize") and the expanded key size (of type size_t) to the parameter list of our function. The prototype looks like the following:

While implementing the function, I try to follow the details in the theoretical list as close as possible. As I already explained, since several parts of the code are repeated, I'll try to get rid of the code repetition and use conditions to see when I need to use a certain operation.

As you can see, I never use inner loops to repeat an operation, the only inner loops are to iterate over the 4 parts of the temporary array t. I use the modulo operator to check if I need to apply the operation:

if(currentSize % size == 0): whenever we have have created n bytes of expandedKey (where n is the cipherkey size), we run the key expansion core once

if(size == SIZE_32 && ((currentSize % size) == 16)): if we are expanding an 32-bit cipherkey and if we have already generated 16 bytes (as I explained above, in the 32-bit version we run the first loop only 3 times, which generates 12 bytes + the 4 bytes from the core), we add one additional S-Box substitution

Implementation: Using the Key Expansion

Finally, we can test our newly created key expansion. I won't calculate the expandedKey size just yet but rather give it a fixed value (the calculation requires the number of rounds which isn't needed at this point). Here's the code that would expand a given cipher key:

Excellent! That post about the Enigma Machine last week, peaked my curiosity.

Throughout the pre-industrial age, I have learned, since communications involved messengers who could always be easily intercepted (for the most part), and not perfectly trusted, the art of "secret writing" was quite important.

You rant and rave about it, but at the end of the day, it doesn't matter if people use it as long as you don't see.
People are free to read the arguments, but if the only way for you to discover gravity is by jumping off a cliff, then that is what you're going to have to experience for yourself.
Eventually, this "fast and loose" approach of yours will bite you one too many times, then you'll figure out the correct way to do things. - Salem

One has to notice that there exist certain cipher that don't need a key at all. A famous example is ROT13 (abbreviation from Rotation 13), a simple Caesar-cipher that obscures text by replacing each letter with the letter thirteen places down in the alphabet. Since our alphabet has 26 characters, it is enough to encrypt the ciphertext again to retrieve the original message.

umm rot13 has a key.... it is a implied key of 13. Sure it can be done with an algorithm. It is a reciprocal ceazar cipher.

"Simplicity does not precede complexity, but follows it." -- Alan Perlis
"Testing can only prove the presence of bugs, not their absence." -- Edsger Dijkstra
"The only real mistake is the one from which we learn nothing." -- John Powell

I would like to let you know that I am still working on this tutorial. I converted it to an HTML version that I hosted on my website. You can find the latest version on http://www.progressive-coding.com/. During the last days, I update some of the features of my website as well, adding a very handy print version of each document and the possibility to download the PDF version (coming soon, couldn't find an html2pdf converter that had full css support).

I'm giving a basic lecture about algorithm complexity tomorrow and will put the document on my site as well.

New update: Today I implemented 2 mode of operations, one that uses only encryption and transforms the block cipher into a streaming cipher and the other a classical block cipher. I'll continue working on the code and put the next chapter in the tutorial up tomorrow.

29th March 2007, 13:08 (GMT +1): I uploaded the new version of the tutorial, it includes the new chapter about the entire AES encryption. You should now be able to encrypt a plaintext of 128 bits. I also added the printer friendly version as well as the PDF version:

3rd April 2007, 11:41 (GMT +1): New version of the tutorial online. It concludes the AES implementation by adding the decryption section. The next and last part will cover modes of operation to use the AES algorithm to encrypt messages of any size.

I am pleased to announce that my AES tutorial is finished. As I explained on my website, I decided to split the modes of operation tutorial into its very own tutorial. I hope you enjoy reading both tutorials and that you manage to implement your very own version of AES to encrypt messages of any size.
Here are the links to the modes of operation tutorial:

This thread is currently the 5th result in google for "Advanced Encryption Standard tutorial", which is kinda great. Since I can't edit my first post anymore, I would like to ask a mod to edit my first post and add a header message saying that the latest version of this tutorial can be found here.