What is FIBRE?

The Fast Internet Bitcoin Relay Engine (FIBRE) is a protocol and implementation designed to relay blocks within a network of nodes with almost no delay beyond the speed of light through fiber. Its design is based on several years' experience operating and studying the Bitcoin Relay Network and functions incredibly well even when faced with suboptimal internet conditions.

FIBRE is designed to be easy to operate for anyone already running a network of Bitcoin Core instances, instantly providing high-speed transfer of blocks. This way, it will significantly decentralize the availability of high-speed relay networks, making it easy for any individual or group to set one up.

FIBRE improves on the design of the Bitcoin Relay Network primarily in two ways: it eliminates latency spikes by sending extra data to compensate for packet loss and is based on the compression provided by the Compact Block work in Bitcoin Core.

Instant high speed block transfer

Because TCP is designed to provide reliable transmission at reasonable bandwidth across medium-large amounts of data, it is incredibly bad at low-latency relay of small amounts of data. It is generally tuned to send packets (each just under 1500 bytes) once and to only discover that some packets were lost after getting a response from the other side. Only then will the sender retransmit the lost packets, allowing the receiver to (potentially) reconstruct the original transmission.

I have seen packet loss over long-haul links on the internet average 1%, though if you purchase bandwidth directly from global carriers, you can expect < 0.1%. At 1%, the probability of transmitting a full, uncompressed, block without needing to wait for extra round-trips is roughly 0.99^(1000000/1500) = 0.1% (or 0.999^(1000000/1500) = 51%, if you're willing to pay for it). Worse still, packet loss starts to go up as you send more data. Even transmitting just 15KB (around 10 packets) has a probability of success at only 90% for an average hosting provider. When we're talking about 1Gbps or 100Mbps links with round-trip latencies of 100-200 milliseconds, a single round-trip takes significantly longer than sending any reasonable amount of data could possibly take.

Thus, in order to have minimal latency block transmission, we must avoid the need for retransmissions at all costs. In order to do so, we must transmit enough extra data that the receiving peer can reconstruct the entire block even though some packets were lost on the way. Luckily this is a well-researched field, largely thanks to video conferencing having similar requirements. The common solution is UDP-based transmission with some relatively simple linear algebra to send data which can fill in gaps of lost packets efficiently (this is called Forward Error Correction, or FEC).

Even with an elegant solution to the packet-loss issue, the time to transmit 1MB over a 1Gbps link is still several milliseconds, which is more than doubled with the extra overhead from both the construction and transmission time of the FEC data. Thus, the Compact Blocks scheme in Bitcoin Core is critical to performance.

Because the Compact Block design was being done simultaneously with the work on FIBRE, the cmpctblock message format was designed to ensure it fits neatly into a UDP-FEC-based relay mechanism. The only difference is that we send it over UDP with FEC, and instead of relying on a round-trip to request any missing transactions in our mempool, we send the block's transactions immediately, again with additional FEC.

Servers relay individual packets to their peers as soon as they arrive.

This way, extra hops do not introduce more latency. Sadly, due to the nature of our FEC encoding, we cannot know if individual packets are a part of a legitimate, or any, block, and thus only enable this optimization between nodes run by the same group.

Several pools already optimize their block propagation using FIBRE

In order to maximize the effect FIBRE has on Bitcoin relay decentralization, I have posted a setup guide for running your own relay network which covers everything from the organization of the software to how to select hosting providers to minimize latency around the globe while keeping costs minimal.