Distributed Systems are really important when building large systems
They can be really confusing – complicated ideas – not well separated out
A story of Two Systems:
SKS (he wrote)
Public Key Encryption (OpenPGP Database)
Replicated, robust, queryable store
Every node should know all of the keys
PKS was precursor to SKS

Why Impossibility Results?
So you know what’s impossible :p
Under the following set of assumptions: The following thing is not possible

“Proving impossibility results causes us to take a very analytical approach to understanding the area. It causes us to state carefully…”
Quote From: A Hundred Impossibility Proofs for Distributed Systems

Zookeeper invented their own protocol. Hard to get right initially – why did they do this:
Programmers usually like to come up with new protocol – it’s fun
Usually with the point of optimizations & customizations
Paper: The Part Time Parliament – Described the original Paxos Algorithm
Paxos Made Simple & Paxos Made Moderately Complex – two papers written on this
Whole industry trying to explain Paxos
Other people put out a Raft Paper
Raft is an easier to understand Paxos
People who use Paxos always have to put in extra optimizations

Ron Minskey – Practical Set Reconciliation

Jane Street – focuses on Consistency?
That’s correct – we’re not concerned with being clever
When things fail – it’s often tolerable to roll back a bit & use a lot of the State
State that you care about tends to be kept by other players

NASDAQ is totally built on State Exchange Reconciliation
When it fails, A Human has to come in & restore it from a backup
That single Leader processors 3 million transactions per second
Actually runs on Java – so Java *can* go fast
Could be done in C or NoCamal(?)