Spread your databases out, into “roles” – roles that you never need to join between ● different users ● or accept you'll have to join in app Each user assigned to a cluster number Each cluster has multiple machines – writes self-contained in cluster (writing to 2-3 machines, not 6)

alternatives at time were either: – closed, non-existent, expensive, in development, complicated, ... – scary/impossible when it came to data recovery ● new/uncommon/ unstudied on-disk formats because it was easy – initial version = 1 weekend