Mathematics , Microcode, Manufacturing, and Musings

Continuing with the back-to-front dynamic compiler, in this post we will:

Use Stack to manage the growing project

Write an interpreter whose result values should be the same as the dynamic compiler

Use QuickCheck to generate tests by comparing the interpreter to the compiler

> stack new compiler

Now we can copy the Haskell file of our budding dynamic compiler into compiler/app, and build using stack:

> cp Main.hs compiler/app
> cd compiler
> stack build

And execute it using:

> stack exec compiler-exe
mprotect = 0
errno = 0
42

QuickCheck

As we build out the compiler, it would be nice to know if its working as intended, not just on a few hand-picked test cases. Writing additional test cases is always good, but I want something stronger — I want to enforce properties on the resulting system. Hand-selecting test cases which test these properties hides the intent, why not just check the program for these properties directly?

Proving properties about programs is hard, sometimes impossible, but we can test individual cases easily. QuickCheck provides a middle ground: state the properties directly, then use random sampling to generate test cases to check. Failure of the QuickCheck property guarantees our property has failed, but passing QuickCheck does not prove the property.

This technique can be used to prove the property too, but it is inefficient in all but the simplest cases: generate all possible inputs to the QuickCheck property and test that it holds for all of them. While this does not work for complex data structures, there is a theory that most bugs occur at (relatively) small input sizes. This leads to another tool, SmallCheck, that does exactly this. SmallCheck generates all possible inputs up to some depth, checking the property for all of them. QuickCheck and SmallCheck together offer strong evidence that your property is held.

Properties

Properties are statements about some set of inputs to the program, and the program’s expected behavior given these inputs. For example, the function “reverse” that reverses a list might be expected to hold the following properties:

reverse [] = []

reverse (a:as) = (reverse as) : a

reverse (reverse a) = a

These look remarkably like unit tests, but where unit tests would have example values, properties have variables. QuickCheck allows any arbitrary predicate as a property, so you are not limited to simple equalities like these.

In order for QuickCheck to test program properties, it will first need to know how to build a program. Recall our trivial expression datatype Expr from last time:

data Expr
= Fixnum Word32
deriving (Show)

QuickCheck uses the type class ‘Arbitrary’ to generate an arbitrary instance of some datatype. The simplest implementation for our Expr type would be:

instance Arbitrary Expr where
arbitrary = liftM Fixnum arbitrary

Here, we are defining the function named ‘arbitrary’ for the type class Arbitrary which generates an Expr containing a Fixnum constructor over an arbitrary Word32 value. Basically, we’re just picking a random 32-bit fixnum and returning it.

While this implementation will work fine, there are so many different 32-bit numbers that we may never really exercise the important parts of the data space. It would be nice, for instance, to ensure we hit all the byte-level crossover points, such as the largest 7-bit number (127) and its immediate successor which requires 8-bits (128). Here’s a list of important test cases:

But we don’t want to ONLY test these few cases — there may be something lurking in our code that trips us up! So let’s rewrite the Arbitrary type class to include our test cases and the fully-random version we wrote previously:

This says we want some of the QuickCheck tests to select a random test case from our list of specific test cases, while most of the time we want completely random Exprs.

As the Expr datatype becomes more complex, these generators will also need to evolve so we can focus on the most important aspects of our datatype, while retaining some truly random inputs to keep us honest.

Program Properties

Now that we can generate a random program (though they are all trivial at this stage), it is time to examine what properties we want to ensure from our compiler. A compiler is supposed to transform a text representation of the program into a machine-executable version that “does the same thing” as the composition of semantic elements described by the language and composed by the programmer.

So, one property we might want to check is “does this program do what the spec says it should?” There is a bit of a chicken-and-egg situation here, since the spec itself is also just text and is no easier to check for correctness than the program.

We can get around this by implementing the language spec as an interpreter. Each operation of the language is written out as operations against a machine model. A program in the language can then be passed to the interpreter for execution, the result of which should be the same as if we had executed the program natively on the hardware.

We’ll start by choosing the type of the interpreter:

evaluate :: Expr -> Value

Our interpreter will be a function from our expression type Expr to a type of values. Since we only have one type of value right now, the next step is easy:

data Value
= VWord32 Word32
deriving (Show)

The ‘evaluate’ function will have a case for each branch of Expr, performing the operations of the Expr on the internal machine model. For now there is no machine model, so the ‘evaluate’ function itself is trivial too:

evaluate :: Expr -> Value
evaluate (Fixnum x) = VWord32 x

This says: to evaluate an expression “(Fixnum x)”, we return the constant 32-bit value ‘x’.

Compiler vs. Interpreter

Back to the QuickCheck properties! We now have two different implementations which should both compute the same answer; the compiler and the interpreter. This makes writing a QuickCheck property relatively simple. QuickCheck will generate an arbitrary program for us, then we will pass the program to both the compiler and interpreter for execution, collect the two results and compare them for any differences. We would like to ensure they are always exactly the same.

This function does exactly as we have described: it receives an arbitrary Expr, calls the interpreter on it, receiving a result value ‘b’, then calls the dynamic compiler to compile and execute the same program natively, collecting the result ‘a’ from it.

The final line tells QuickCheck that we want these values to be equal.

Note that this property is specialized to handle return values which are 32-bit integers. This will need to be generalized as we add more advanced data types to the language.

The main program simply allocates a buffer for dynamic code, and calls QuickCheck to fuzz our compiler.

Next Steps

We have completed the first step of the compiler as described in “An Incremental Approach to Compiler Construction”, which we began in part 2 of this series.

Next time (part 5) we will add more immediate value types such as boolean and character, requiring modification of our infrastructure and QuickCheck properties. In particular we need to choose a representation for tagging or ‘boxing’ our values to differentiate an integer from a boolean.

This is the second post in Paxos Made Complete, a series highlighting design decisions when building a system based on Paxos and the State Machine Approach. Code for the series is available in the github repository. Today we will finish the Classic Paxos implementation by embedding it into a small simulator, then generating tests using QuickCheck.

Fuzzing with Event Streams

Critical infrastructure requires thorough testing, and distributed computing is an especially challenging domain to test. The technique I have found most valuable is deterministic simulations driven by a generated event stream. For small networks, I can simulate a complete multi-node system on a single machine and fuzz it with a stream of random events. The deterministic nature of the simulator allows me to write the event stream to a file and replay it later for regression testing.

For Paxos, this means defining some interesting events which can occur in the real world, then writing a simulator which accepts a stream of these events and interprets them by calling our Paxos function(s). Testing becomes the process of generating a random stream of events and running them in the simulator, checking assertions all the while.

Since the entire state of the global system is available locally, it is trivial to check global properties between each event, ensuring correctness at all times. Another benefit to this technique is the exact event which causes the inconsistency is discovered immediately.

For this blog series, I chose a set of events which is sufficiently interesting to expose weaknesses in the protocol, but not so many as to be a burden to implement. Here’s what I chose:

Deliver, Drop, and Duplicate should be self-explanatory. Shift rearranges the message ordering to simulate packets delayed in the network.

Tick and Req may require a bit more explanation:

Tick Event

In production Paxos environments, there must be a signal to cause a Leader to send the Prepare message. This signal is often derived from a Leader Election protocol. The leader election is driven by timeouts and other inputs which combine to form a suspicion of node failure, causing the election process to commence.

The Tick event skips all of that – we declare that this node believes itself to be the Leader and will attempt to gain control of the voting. This is a conservative strategy; all possible leader elections will be covered by placing the Tick event at the right spot in the event stream, as well as a variety of situations which we hope would not happen in the real world, but which Paxos should handle correctly anyway.

Req Event

In typical implementations there are two entities outside Paxos; a client application which is requesting values to be added to the log, and an under-consensus application consuming the values from the log. For example, a web app sending database requests to the cloud, and the database applying the updates in the order specified by the Paxos log.

The Req event simulates the client’s request to add values to the log. There are so many design decisions related to client semantics, expected behaviors, and control flow, I’ll have to leave that for a future post. For now, we will assume that clients will resend requests if they don’t receive a response for awhile.

QuickCheck

QuickCheck is a library which generates random instances of a data type and feeds them into assertions to check a program’s properties. If it finds a failed assertion, it will automatically trim the datatype, looking for the shortest/smallest example which triggers the assertion. The result is an input to the assertion which causes it to fail — a counter-example to the assertion.

This makes QuickCheck particularly valuable for debugging; it will not just tell you an error has occurred, but offer the actual inputs needed to create the failure. Haskell’s pure functional semantics has the added benefit of being deterministic, so these inputs can be stored as regression test cases without needing the often-complex setup code needed to test imperative programs.

In order to fuzz Paxos with QuickCheck, we must supply a few functions to teach QuickCheck how to walk through our data types and what (interesting) values to insert in each location. Here’s a few examples for the types we defined last time:

Once QuickCheck is setup for our types, it’s time to define the properties we want to check. A property is any deterministic predicate. In Haskell, that means just about any function returning a boolean!

The Consistency Property

The main property of Paxos is its freedom from inconsistency; any correct node will reach the same decision, and decisions never change once made. Our job is to encode this property as a function which takes a list of events and returns TRUE if Paxos held the property, or FALSE if an inconsistency was detected.

My tactic is similar to the method described in “Paxos Made Live”; execute random events against a fresh network, simulating an unstable period, then run a series of “stabilizing” events to flush out the protocol. If the nodes disagree on the result, an inconsistency has occurred. If we carefully select the stabilizing events such that they are the very minimum possible events to complete the protocol, we can also detect “live-lock” situations (since any remaining undecided nodes would be “out of spec”).

I’ll skip most of the infrastructure needed to get the simulator running. Here are the important parts:

The function “prop_ConsensusAchieved” is the property we want to test. It takes a list of Events (generated by QuickCheck), executes the events using handleEvent on a Network containing simulated nodes and a message queue. Once the “unstable” network has run, we offer a stream of “stabilizing” events, after which we extract values and compare. If this leaves us with an inconsistency or no agreed value, the property has failed.

Design Decisions

I’ve used a technique often described in Paxos literature where all outgoing messages are also sent to our self for processing. This allows the logic for each Role to remain separate and avoid code pollution and duplication.

The handleEvent function hides one important feature; messages sent to our self are always delivered. This overrides the Drop, Duplicate, and Shift events. As written, this still consumes an event from the queue. A better technique would be to immediately deliver the message to our self, although this poses the risk for an infinite loop of self-delivery…

The selection of events is somewhat arbitrary. Specifically Duplicate and Shift, which I wanted to be deterministic even if the intervening events were to generate a different number of output messages into the network queue. The semantics I selected meet my criteria, but I’m not totally satisfied with them.

The design presented here is slightly more complex than necessary in preparation for an upcoming step which will allow us to test multiple different versions of Paxos against each other on the same event stream(s).

Next Steps

Next time we will modify Classic Paxos into Mult-Paxos, and it’ll blow your mind just how trivial the changes are! From there we will begin examining efficiency by optimizing the structures and message formats to streamline the trivial implementation.

Back in 2007 I collected some notes I had written on Paxos over the years into a survey paper and published it to Wikipedia under the Paxos Protocol article. In the years since, there has been an explosion of Paxos-related papers, blogs, and projects. I am honored to find quotes, graphs and commentary on my article as I read through the new papers and projects in fault-tolerant distributed computing.

Unfortunately, only the most basic discussion of the protocol and system-related design decisions seem to be covered. As an expert in the protocol and its implementation, I find the lack of community discussion regarding advanced, systemic uses of Paxos concerning.

I hope this blog series will help correct that. I will build a fault-tolerant distributed execution engine for the state machine approach, covering all aspects of the craft. Each article will add a small step, describing the tradeoffs and selecting one of the many paths available on the way to a complete solution.

Paxos Made Trivial

In order to set the stage for this series, we will need a Classic Paxos implementation with nothing but the bare essentials. I’ll be writing in Haskell, but practitioners of the art should recognize the patterns I’m using and find easy translation to other languages. I’m not using any Haskell magic — just basic data structures and simple functions such as less-than, max, and increment.

If you’re new to Paxos, I recommend my Wikipedia article, or any of the other papers, blogs, etc. which cover the core protocol (Google it!).

Let’s get started!

There are only a few data types used in Paxos:

We need a way to tell two nodes apart. Let’s call this a NodeID. It’s generally represented as an Integer. NodeIDs are unique and totally ordered.

We need a way to specify what we want to agree on. Let’s call this a Value. We’ll ignore the content of the Value for now (it’s irrelevant), but it is often a Byte Array. Additionally, Values need to be compared to each other for equality, and there’s a couple of special Values we need; NoValue and NoOp.

Finally, we need a way to tell which Value is currently winning. This is called a Round in the literature, and may be most easily expressed as a pair of an Integer and a NodeID. Rounds are compared lexicographically; the Integer first, then the NodeID.

Every Value is bound to a Round, and they must never be separated or changed! We’ll call this a Vote.

This says “A NodeID can be the value ‘NodeA’ or the value ‘NodeB’ or …”. Likewise for the Value type. For the Round, it says “A Round is a data structure constructed using the reserved name ‘Round’ followed by an Integer and a Node ID”. The ‘deriving’ clause tells the compiler to write a couple of useful functions for us, in this case equivalence and ordering.

I’m using enumerations for the NodeIDs and Values to reduce the state space in this example code. This will allow us to test deeper into the protocol when we build the test framework. It should be clear that these may contain arbitrarily complex data as needed for the system you are building. Typical choices might be IP addresses for NodeIDs, and byte arrays for the Values.

That’s it! Four very simple data types. These might be classes or structs in other languages, but they would look about the same.

Roles : Leader, Acceptor, Learner

There are three distinct Roles in Paxos. While these may be combined in various ways, it will help the discussion to keep them separate for now.

You should note the lack of complexity here — each Role holds just a few variables! More advanced versions of Paxos add just a fraction more complexity to this basic form.

They all keep one Vote, although its meaning differs slightly between the Roles. The Leader must also maintain its latest Prepared round and a quorum of Promises. The Acceptor just keeps its latest Promise. The Learner needs a quorum to track Votes.

I want to point out that this is essentially the minimal set of data required for any system which could solve distributed consensus. It could be combined or split along different lines, but every unit serves a necessary purpose in the protocol — there is no cruft.

I won’t go over each of these, as they are thoroughly covered in the literature. I’ve given (arguably) readable names to the fields so we can distinguish them in the code.

Protocol

Now we reach the meat of the protocol. In the following, we will define a function for each role. The input is the current state and an input message. The output is the new state and an output message.

Acceptors receive the Prepare message and always send a reply, but only Promise to equal-or-higher round numbers. This reply serves as both the regular Promise message, and also a Negative-Ack to other Leaders of lower rounds.

Upon receiving a Promise, the Leader will ignores Promises for older rounds. If the promise matches the prepare this leader sent, it adds the Acceptor to the Quorum. If the promise is for some higher round, the leader abandons the current round and records the higher round. Along the way, the vote from the highest round is selected as the value to propose next.

At some point (possibly immediately after the previous message), the system tries to get a Value agreed. We simulate this by sending a Request message to the Leader. If the Leader has a quorum, then it selects a value to propose. If a value already exists, as discovered in the Promise messages, we must choose it. If not, the Leader can propose any value (ie: the one being offered in the Request).

There you have it; about 100 lines to implement the complete protocol!

Design Decisions

Many papers and blogs discuss the need for a new message in the protocol, the Nak message, to be returned when a leader’s prepare round is too low. This situation is never necessary. The protocol already has a message perfectly suited for this (Promise).

The paper “Paxos Made Live” describes a situation in which a replica behind a partition continuously increments the round number trying to gain leadership, only to cause immediate disruption of a stable group when the partition is repaired. Here we see this situation never needs to occur. The Leader-to-be simply does not increment its round number on timeout. It only increments the round number when receiving a Promise to a higher round from an Acceptor. This eliminates the need for the added complexity of “bumping” the round number at regular intervals to avoid the disruption, as well as the necessary tuning of this new parameter.

When a Leader receives a Request, but is forced to choose a different value, what happens to the Request? The answer to this question significantly affects how the client must behave, and what expectations it has of the distributed system. In the code above, the request is silently discarded. Most implementors would choose to return a “retry-later” message, or queue the request internally.

How do we know when a decision has been made? There are many different ways to count a Quorum, such as majority-rules and weighted votes. Most Paxos implementations require a majority of votes from Acceptors to achieve a Quorum.

Conclusion

I hope this post has set the stage: Paxos is a small, simple protocol for solving a difficult distributed computing problem. In roughly 100 lines of Haskell, we’ve defined all the core types, data structures for each Role, all the messages used, and implemented the basic protocol.

Next we will fill out some infrastructure for testing, then look at adding some features which bring the implementation closer to real-world needs.

I’m just a “regular” software engineer. So when I attended a recent conference on functional programming, my reasons for attending were often remarked upon. I was asked to offer some insight into how I ended up in the land of Functional Programming, and what lessons I might impart about my journey.

To set the stage; I’ve been programming from a young age, and graduated college having majored in Computer Science. I’ve used every major language in that time for personal projects, and almost all of them in professional projects. A sampling of languages I had used before this story includes: Pascal, C/C++, C#, Java, and Objective C.

You may infer that all of my formative experiences and all of my formal training focused solely on procedural and object-oriented languages. At the time of this story, I had complete confidence in these tools and my skills to solve any software engineering challenge that might arise. But in spite of this preparedness, it was the very tools I was taught to use that would betray me.

In high school I was an avid logic-puzzle solver, voraciously scouring the school and public library for new conquests. Along the way, I came across the “Mathematical Recreations” column in Scientific American. In particular, one article described “Core War,” a game where programs written by the players fought against each other in the memory of a virtual computer system. The winner was the program that caused the enemy to execute an illegal instruction.

The language of Core War is called Redcode, a simplified version of assembly language, and served as my introduction to many advanced concepts including registers and addressing modes, data vs. text segments, multi-threaded concurrency, defensive programming (literally!), virtual machines, and low-level debuggers. Truly, Core War was one of my foundational learning experiences!

I played the game for many years, contributing several articles to The Core War Newsletter (Vol 6 issues 2 & 3). The evolution of the language itself and the strategies involved in winning offered a continuous stream of intellectual and technical challenges. The apex of my Core War experience was writing a genetic algorithm to evolve warriors and pitting them against the latest-and-greatest warriors in the competitive hills.

The first ‘real’ language I tried learning was C. I had written extensively in BASIC throughout middle school, but never felt satisfied. It was too far removed from the hardware, too distant from the ‘true’ nature of the computer. C, on the other hand, seemed like a backdoor left open by some inattentive operator, with direct access to the control room.

I purchased a copy of “C – The Complete Reference” at a local bookstore, my first technical manual.

I believed the tome to be an ancient wizard’s codex containing all the knowledge needed to master C programming, and by extension the computer. I was unprepared for the depth of technical background and vocabulary required to put the book to use. The arcane, almost alien, language consisted of symbols jumbled in random order, indirect references to objects of unknown identity lost in translation to the English language.

Simply put, “C – The Complete Reference” was not a good introductory programming guide for middle school students.

I turned instead to Pascal, the ‘native’ language for my new computer – a Macintosh 512K aka “fat mac.” Apple’s API toolbox, the ROM itself, used the Pascal calling conventions. It would be some years before I returned to C, this time with little doubt I could master it.

I’ve been a programmer for almost my entire life – I still remember my Dad letting us play games on his early home computer. I was in first grade in the early-80s and “home computer” was an infrequent term. I was playing a game not unlike Missile Command, becoming frustrated that the enemies were too hard to shoot.

I wanted them to move slower. More specifically, I wanted my Dad to make them move slower. In the naivety of youth this seemed trivial, something my all-powerful father could certainly accomplish with just a wave of his hand. Of course I was distraught to discover that he could not alter the game’s behavior.

This led to a different kind of frustration, one which launched my career and still drives me today; HOW do those little dots move around in the first place? What controls them, and how can *I* become their master?