Accidental Hardfork – The Resilience of a Fledgling Bitcoin

In the early hours of March 12th 2013 the bitcoin currency and community was tested in a big way. What resulted is a testament to the resilience of bitcoin, and has given me personally substantial confidence in bitcoin.

What created the problem:

Version 0.7 of the bitcoin client used Berkley DB to store and process the blockchain. However BDB was proving slow and version 0.8 of the bitcoin client moved away from BDB for a much faster alternative LevelDB.

However unbeknownst to anybody, there was a bug within BDB which caused the version 0.7 clients to fail processing blocks of a certain ilk (the specifics are uncertain but unimportant, however suspected to be relating to a high number of transactions) – however since all version 0.7 clients had the same problem, there was never an opportunity for another client to gain majority and process these otherwise erroneous blocks.

When, however, the majority of network hashing power moved to the version 0.8 software, a hard fork occurred when one such erroneous block was created by the 0.8 block chain (to be clear, it isn’t actually an error with the block, only in how v0.7 deals with a block of this nature) with the 0.8 block chain growing faster than the version 0.7 blockchain, but since the 0.7 blockchain couldn’t validate this particular block (and all blocks henceforth), it continued to process the 0.7 blockchain believing it was the only true and correct blockchain.

The Temporary Solution:

Proposed within 15 minutes of the problem being identified, the temporary solution was to roll back the majority of hashing power to the 0.7 client and accelerate the 0.7 blockchain hashing power until it exceeded the 0.8 blockchain, at which point the two blockchains would converge.

The affect on bitcoin prices:

Whenever news of this nature comes out a few things happen.

First of all those with the information early sell off a bunch of bitcoins because they know a drop in price is imminent.

Second, when the news hits the masses, not understanding what is really going on they dump theirs coins and run.

Finally, those who know and understand what’s going on capitalise on the weak will and ignorance of the cut and runners, buy back in, and make a tidy profit.

On two occasions recently the price of bitcoin has been tested severely.

Bitcoin Price Event 1:

On the evening of the 6th/7th of march transaction volume on exchanges was extremely high, causing the exchanges to lag behind on displaying accurate price figures. At the peak this delay was in the order of 16 minutes behind.

The price per bitcoin had rocketed recently, and was nearing weak resistance at $50 per BTC – the feeding frenzy was on. But then when the exchanges started lagging behind, people started to panic because they couldn’t see accurate figures so they sold their coins. This began a fall in the price per bitcoin, and with “2011 bubble” ringing in peoples ears, a panic sell ensued.

Prices fell from above $49 to around $44 over the course of a few hours, and then further to £33 within about an hour.

When this data started coming through to people, a mass buy followed and the price immediately recovered to about $43.

This itself is an incredible event for the strength of bitcoins. To be able to suffer such a huge massive panic sell, only to recover immediately, is a true testament to the value of bitcoin as a currency and not just a speculative investment.

Bitcoin Price Event 2:

The second price event happened as a direct result of the accidental hardfork event described above. In this circumstance the price drop was based on ignorant panic, which took the price down to $36 in under an hour, but which then immediately sprung back to $46.

Having a second such event so closely following the first I believe is an incredibly strong sign of support for the current bitcoin price, which I fully expect will soon surpass $50 and continue on its journey of meteoric rise.

Blow by Blow:

Below is a blow by blow account of the problems surrounding the hard fork, and how quickly they were addressed. By anybody’s standards this was an incredible display. Along with the two price events described above, the resilience of the bitcoin economy is evident.

Compare if you will, the issues faced by bitcoin during this event, with the issues faced by major banks on an all too regular basis. While a bank would pontificate and wollow in inaction while blame was being passed up the corporate chain, and then back down the corporate chain, the bitcoin developers identified the problem and the fix within 20 minutes.

Although it took a further 6 hours for the fix to win out, a 6 hour partial downtime is one incredible response to what could have become a major issue.

11:30pm problem starts to come to light with one user struggling with their version 0.7 bitcoin client.00:00 The problem didn’t really come to light however until about midnight, when a couple of devs starting noticing something was up.00:05 It took a mere 5 minutes for “Luke-Jr” to correctly identify the hardfork with the message “Luke-Jr: so??? Yay accidental hardfork? ”00:08 Just three minutes after Luke-Jr first identified the issue, user “sipa” tracks the problem to a bug (as yet unidentified) in the version 0.7 software. Also within three minutes The MtGox Bitcoin Exchange confirmed it was on Version 0.7 “old” software.

At this point Luke-Jr raises a concern “my immediate concern is, which fork are the majority of miners on?” which is echoed for the next few hours.

00:11 The Mt.Gox exchange stops accepting bitcoin transactions (concerned they could be double spent)00:20 a few people start to freak out, and messages such as “omg what is going on” and “oh fuck” begin to appear in the chat.

But – in the same minutes (less than 15 minutes since the hardfork was identified) Luke-Jr (aka “hero for the evening”) suggests what will eventually become the short term fix, saying to a fellow dev “sipa: seems to me the best short-term fix is to get miners off 0.8”00:22 merely two minutes after this fix was suggested a discussion begins between the devs on whether or not they can reach a consensus on this.

This is when lead developed Gavin Andresen joins the conversation and is initially very resistant to the proposal.00:31 while a consensus has not yet been reached, major mining pool BTC Guild begins to take action on rolling back their software to version 0.700:39 Gavin Andresen expresses strong resistance to the proposed solution “Going backwards is the wrong thing to do, in my opinion”00:43 the bitcoin price on MtGox begins to suffer dropping $2 to $46/BTC00:45 Despite Gavin Andresens initial protest, the devs eventually reach consensus when BTC Guild announces they have the balance of hashing power and all agree to request pools downgrade to version 0.7 as a matter of urgency.00:46 A bitcoin gambling website which has been at the brunt of a lot of blame, Satoshi Dice, comes to the party and pauses all transaction processing, lightening the load on the network.00:50 the bitparking mining pool begins downgrading their software to version 0.701:15 the Mt Red mining pool shuts down its services as a quick solution to reduce hashing power on the version 0.8 blockchain.01:19 Major bitcoin payment processer BitPay intentially freezes transactions to reduce the impact of a potential (but unlikely) double spend attack01:23 Although struggling through a number of issues surrounding moving large files between servers, BTC Guild mining pool finally downgrades the first of its servers.02:36 the BTC Guild mining pool downgrade is completed across all of its servers.03:00 A temporary bandaid patch for software version 0.8 is release which should allow it to rejoin the 0.7 blockchain and ditch the problematic 0.8 blockchain06:21 Everyones hard work is reworded as the fix proposed at 00:20 dominates the blockchain and the version 0.8 blockchain merges once more with the version 0.7 blockchain06:23 The Mt Gox exchange announced they would wait until a further 6 block confirmations came through to ensure there was a clear winning/leading chain. This wasn’t necessary, but a cautious safeguard.

Conclusion:

While I don’t doubt at some point another accidental hardfork event might take place (although this was the first in over 225,000 blocks and 4 years), If such an event were to take place I would sleep easy knowing the bitcoin developers and community have things in hand.

My only other takeaway from this situation is a slight loss of faith in the lead developer of bitcoin, Gavin Andresen, who in my mind could have acted quicker and more decisively, and could in theory have lessened the impact by a matter of hours.

My own view is that Luke-Jr was the star of the evening (although he assumed the problem was with 0.8 software, not with 0.7 software, which was an easy assumption to come to in the heat of the moment) – Sipa was also a stand out contributor of the evening, taking an equally strong role in the recovery and working extremely effectively with Luke-Jr. In my mind these two deserve the praise.

I do wonder if there is scope for the creation of a centralised (although voluntary) notification system for the key players and network at large to close ranks more quickly in any future event such as this one, but I also think this would need to be managed with a very level head. In my mind last nights even should have been an amber warning event, not a panic event, as the solution was clear, simple and quick.

One honest bitcoin user has reported he was able to double spend $10,000 worth of bitcoin, but I believe this is more down to the payment processors flaws than it is the network and bitcoin protocol.

4 thoughts on “Accidental Hardfork – The Resilience of a Fledgling Bitcoin”

Bitcoin promised non-reversible transactions. OKPay gave a customer the ability to withdraw $10K USD after the bitcoin transaction for deposit at OKPay reached six confirmations. Then that customer, about an hour later, broadcast a double-spend transaction that was successful.

The blockchains did not and cannot merge. The 0.8 fork was abandoned when the 0.7 fork became longer than the 0.8 fork, because the protocol specifies that the correct fork is the longest one in existence.

When the 0.7 fork became longer, all 0.8 clients and miners automatically switched to that fork.

Donut, I believe I was referencing the merging of hashing power/miners across the two versions of the software. Although you are entirely correct.

Anonymous – it makes sense that a payment provider should have checks and safeguards in place to detect if there is a hardfork at which point its service should pause. That would be a smart approach to avoiding this problem in future.

I do wonder if there is opportunity for the development of a centralised (although voluntary) alert program for the key gamers and program at huge to shut rankings more easily in any upcoming occasion such as this one, but I also think this would need to be handled with a very stage go. In my thoughts last night time even should have been an ruby caution occasion, not a anxiety occasion, as the remedy was obvious, easy and easy.