znProjects Blog

Software is beautiful again!

MD5 in F#, Functionally

The Advent of Code Day 5 problem presented a very interesting challenge - it required an MD5 hash function to get the final answer.

As my goal with the AOC challenges is to increase my knowledge of F#, I chose to:

Implement my own version of the MD5 algorithm

Implement it functionally (no mutation, if possible)

Out-of-the-box implementation

.NET has an out-of-the-box implementation of MD5, which can be accessed quite easily from F#. It resides in the System.Security.Cryptography namespace. Here is a function that takes in a string and outputs its hash as a string.

It makes it very easy to check whether my hash function is working correctly because its output must match the OOTB MD5 algorithm's output.

It provides a baseline against which to compare performance.

Planning the implementation

First, let me just say that I have never implemented a "cryptographic" algorithm, such as a hash function, encryption algorithm, etc. I knew that the implementation would be more difficult than a normal algorithm and so I spent a considerable amount of time reading the following resources.

The first two resources helped me understand the flow of logic. The last resource is a compilation of MD5 implementations collected in one spot and, luckily, written in multiple languages. This gave me different perspectives on how the code can be broken up, alternatives for implementation, etc.

I relied especially heavily on the Java, C#, Python, and Haskell implementations because those are the languages I am most familiar with.

Considerations

The algorithm assumes that all values are stored in the little-endian format.

The algorithm requires a bitwise-left-rotate function, which F# does not provide.

How to store a 128-bit hash.

Initially, and I'm not sure why I thought this, I was under the assumption that Intel CPUs use the big-endian format. However, they actually use little-endian, which makes that consideration irrelevant.

It took me an embarrassing amount of time to hunt down a bug where I forgot that the F# left-shift operator <<< is NOT a left-rotate operator. Once I narrowed down on the bug, it was easy enough to implement a function to perform the rotations. However, this was a good lesson in never assuming anything when reading code - my eyes were just glazing over the <<< operator, even though the problem was staring me in the face.

Finally, I chose to store the values as 4 uint32 integers. This is similar to what the RFC authors, Wikipedia authors, and other implementations did on Rosetta Code. While F# does have the uint64 type, it would have required translating all the 32-bit-based pseudo-code / code (introducing an element of risk) and I'm not sure what benefits, if any, it would have provided.

Mapping from Wikipedia to F#

I started my analysis and design phase by taking the pseudo-code from Wikipedia and mapping it to how I wanted to break up my implementation. I tried to use the same names for variables and, for consistency's sake, I ended up using the Wikipedia naming scheme.

Oddly enough, Wikipedia uses different variable names compared to the RFC pseudo-code and I did not find a good reason for why the original writers did this.

For my solution, I chose to go the function route instead of hard-coding the table into the source code. This was primarily for brevity's sake. A side benefit was that it allowed me to put in a long and oddly satisfying function chain.

This part actually gave me some trouble because I thought that I had to add one bit (containing 1) to my message in the MSB (most significant bit) slot, and then start appending 0s or the message length immediately after that. That turns out not to be the case, and reading the RFC and other implementations on Rosetta Code provided a good solution to the problem.

Essentially, following the message, we need to add a 0x80 16-bit word followed by the length (restricted to 8 bytes). The intervening space, if any, is filled with 0 bytes.

This is the one part of the implementation that had the highest potential to be "impure". Specifically, the part that constructs the padding. However, F# array comprehensions removed the need to mutate an array to construct the padding before it is appended to the msg.

Step 7: The core of the main loop (Yes, I know this is out of order)

The Wikipedia pseudo-code is written in an imperative manner where the main loops are encountered before we get to the meat of the hash construction algorithm. However, in order to explain my construction, I will start by looking at the implementation for a single iteration of the "Main loop", and then build my way out from there.

In F#, I have three separate constructs that we need to look at. First, I followed the technique used by a few other implementations and collected the functions represented by F in a function list. This allows index-based access to the correct function.

Note that although Wikipedia uses the labels a0, b0, c0, and d0, these are mutable values that permanently change with each 512-bit chunk that is processed. Thus, each new 512-bit chunk starts with these 4 values that have been changed by all the previous chunks that have been processed.

The flow in this part of the code, without the details from Step 7, can be broken down into 3 parts:

Start with an initial value.

Run an algorithm a limited number of times, wherein each execution of the algorithm uses the results from the previous run as the input.

Take the final value and continue with the overall flow.

This is a perfect scenario for a fold operation. Here is the F# code from my implementation.

The rest of the function is devoted to converting the MD5 hash into a user-friendly representation.

Testing

Based on the testing I've done so far, this algorithm works correctly and produces exactly the same results as the OOTB MD5 algorithm.

I first started by implementing the "standard" tests (originally specified in the RFC) and comparing the results with the OOTB algorithm.

Then, I added an FsCheck test that compared my algorithm against the OOTB one for 10,000 tests - all passed without any problems. Now that I've finally learned how to properly use FsCheck at a basic level, I'm starting to find it difficult to completely trust my results unless all the FsCheck tests pass. I don't think that this is necessarily a bad thing, because it tends to make explicit assumptions that I made in my code (e.g. for my MD5 algorithm, I assume that the original message is an actual string and not just a NULL).

Performance

I measured performance by using the AOC Day 5, Part 1 problem, which computes thousands of MD5 hashes. Here are the results, derived using F# interactive's #time directive. I ran each test 3 times and averaged the results.

Average run-time and garbage collection performance of the algorithms, in seconds

Algorithm

Real

CPU

Gen0

Gen1

Gen2

OOTB

53.37

53.37

6780.33

6.00

0.67

Functional

172.79

172.76

34754.00

17.67

1.67

Here is the same chart, but with percentages.

Average run-time and garbage collection performance of the algorithms as a percentage, with the OOTB algorithm as the baseline.

Algorithm

Real

CPU

Gen0

Gen1

Gen2

OOTB

100%

100%

100%

100%

100%

Functional

324%

324%

513%

294%

250%

Obviously, my functional version of this algorithm is a LOT worse than the OOTB implementation.

Lessons Learned

Be extremely careful of assumptions when reading code. Each character and symbol should be analyzed to ensure that it is appropriate for the task at hand.

Define the tee function at the beginning of each project because it WILL be useful at some point.

It's been a goal of mine to implement a "cryptographic" algorithm for a few years now, but I never found the time before. I have to say that it was a lot of fun and I can't wait to do it again!

Not strictly a lesson, unless that lesson is "have fun with hobby projects".