The danger of agile iIncrementalism

It shows up in just about every list of attributes that companies either use to describe themselves or aspire one day to be: agile. The thinking goes something along the lines of:

The technology world is rapidly evolving, from both a technology and a business perspective. If we want to thrive in this environment, we have to be adaptable to change. We have to be agile.

This is typically followed by some reference to the automotive industry (either when it disrupted the horse and carriage businesses, or when the American motor companies were left with gigantic SUVs while others introduced more green alternatives). Truthfully, there is a lot of validity to the calls for agility in a rapidly-changing (or just emerging) market. The need for learning and iteration are undeniable.

But the combination of agility and incrementalism can be a potent recipe for disaster.

Before I get into this next point, I want to make sure I give credit where credit is due. Many people have read The Innovator’s Dilemma by Clayton Christensen. Some will be aware of his follow-on work The Innovator’s Solution (the stronger of the two books, IMO). Christensen co-wrote the latter of the two with a gentleman named Michael Raynor. Raynor went on to write another book, The Strategy Paradox, that (I think, anyway) is the most advanced of the set, and it remains to this day the book that has had the most profound impact on my own thinking.

A central point to Raynor’s thesis is that it is not enough for a company to be adaptable; you have to adapt at the rate of your surroundings. If you are too slow, you are susceptible to disruption (the car analogies). The less intuitive leap, though, is that if you are too fast, you are equally susceptible. I will argue that the big server guys were too slow to react to the server virtualization movement. But is it possible that the network vendors have been too fast?

I’ll summarize a particularly powerful part of the book on what Raynor calls “slow change”:

The Vikings, led by Erik the Red, established a presence in Greenland around 985 AD. With a population that grew to around 5,000 people, they survived for nearly 450 years through a combination of farming, animal husbandry, and trading with Europe. Sometime in the first half of the fifteenth century, the Greenland Norse died off, likely because of starvation. Despite being amidst abundant wildlife, they refused to hunt and fish, and so they died while being surrounded by the food and resources that otherwise could have saved them.

How does this happen?

Shrinking intervals between colder winters and unusually short summers made it increasingly hard to keep their herds. The same weather patterns made it exceedingly difficult to maintain the trade lines along the Northern Atlantic that kept them stocked with European supplies. As the trade ships arrived less and less frequently, the goods were more and more expensive. Without European building supplies, maintaining proper shelter for both the people and the animals was even harder.

The weather changes didn’t take their toll in a single season. Instead, they slowly ate away at the settlements over time. Each year, things got a little bit harder. But the Norse were innovative in their agriculture and trading practices, so by adjusting practices, they could just barely keep pace with the changes they were facing.

Raynor writes the following conclusion:

“Eventually, however, the limits of their ingenuity were reached and the stresses placed on their society by the slowly changing environment overwhelmed them. One drop at a time, the water behind the dam rose, and one sandbag at a time, the Greenland Norse reinforced it. Eventually, however, the dam burst, and when that happened, it was too late to dig a canal. In a classic paradox, during the good times there was no need for change, and by the time it was obvious that radical change was needed, there was no capacity to effect it.”

When you are committed to incrementalism, you become proficient with eking out small gains. As the market or technology changes, vendors become incredibly adept at sliding a few resources over to accommodate the change, Add a feature, extend a protocol, re-spin an ASIC if you have to. On the customer side, tweak a deployment, add a bit of policy magic, segment the network to isolate the pain. All valuable, but all incremental.

With SDN in particular, I for one am interested to see if our industry continues to tweak our way through. Will a little bit of automation magic be enough to placate people who have been suffering from years of longer winters and shorter summers? Or does the true promise of SDN require more than an incremental approach to networking? And if it does, which of us are capable of setting aside our infatuation with agile incrementalism and actually embrace true disruptive change?

In the end, the fact that we are capable of making BGP do unnatural things is good in the short term, but it might be the very thing that prevents our own evolution. Our collective ability to adapt to change might be the one thing that prevents us from actually changing.