BFTSim: A simulation framework for comparing BFT protocols

Much recent work on Byzantine state machine replication focuses on
protocols with improved performance under benign conditions
(LANs, homogeneous replicas, limited crash faults), with relatively
little evaluation under typical, practical conditions (WAN delays,
packet loss, transient disconnection, shared resources). This
makes it difficult for system designers to choose the appropriate
protocol for a real target deployment. Moreover, most
protocol implementations differ in their choice of runtime
environment, crypto library, and transport, hindering direct
protocol comparisons even under similar conditions.

We present a simulation environment for such protocols that combines a
declarative networking system with a robust network simulator.
Protocols can be rapidly implemented from pseudocode in the
high-level declarative language of the former, while network conditions
and (measured) costs of communication packages and crypto primitives can
be plugged into the latter. We show that the resulting simulator
faithfully predicts the performance of
native protocol implementations, both as published and as measured in
our local network.

We use the simulator to compare representative protocols under
identical conditions and rapidly explore the effects of changes in
the costs of crypto operations, workloads, network conditions
and faults. For example, we show that Zyzzyva
outperforms protocols like PBFT and Q/U under most but not all
conditions, indicating that one-size-fits-all protocols may be hard if
not impossible to design in practice.