Networks, Collective Intelligence and the Future of Social Media

Why Do Great Ideas Take So Long to Spread?

Just because a new fact or idea seems right, doesnt mean it will spread like wildfire. Evolution, hand washing in hospitals, the inevitability that personal computers were the future of technology — none of these ideas were accepted immediately, even though they seem obvious today. Change takes time. But why?

The short answer is we’re intellectually stubborn. We don’t always weigh all the evidence before we make a decision, and this is especially true if a change of opinion requires a wholesale overhaul of our worldview. Usually, we’re defensive in the face of change, spouting alternative theories and contradictory data. Although this type of resistance can help keep everyone honest, it can also produce very bad effects.

Just take Ignaz Semmelweis — a physician who recommended doctors clean their hands prior to delivering babies — who was ignored and essentially driven mad by his colleagues refusal to accept the truth. But eventually, in the face of overwhelming evidence, the majority will generally accept the new theory, before their recalcitrance becomes too counterproductive.

Shifting from an old view to a new one is never a clean and seamless process. As numerous scientists have experienced, trying to get a new idea accepted is usually a messy process — and a long one. In fact, it could take until the retirement or death of the holdouts and the influx of younger and more open minds for the new idea to become accepted. The physicist Max Planck seems to have summed up the issue with this maxim: ‘New scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.’

This seems intuitively obvious. Since science and business are human affairs, we can’t expect the old stalwarts to change their minds when a new idea comes along. We just have to wait for them to die. Seems rational enough.

But here’s the thing: Planck’s Principle turns out to be wrong.

Consider Charles Darwin’s theory of evolution. Several decades ago, a study (PDF) examined sixty-seven British scientists from Darwin’s time and found that only about three quarters of them had accepted Darwinian evolution ten years after On the Origin of Species was first published in 1859. So evolution was not the rapid change we thought it might have been.

Why? If events had unfolded according to Plank’s principle, then young scientists would have rapidly accepted Darwin’s ideas while older scientists would have resisted them. But it didn’t happen that way. Although it’s true that those who accepted evolution were younger on average than those who still rejected it after ten years, age explained only 5% of the variation of acceptance or rejection of this theory. The younger scientists didn’t necessarily accept it rapidly; they accepted it at a rate similar to the older scientists who accepted it, over the course of a decade.

So it turns out we can’t even rely on common sense for understanding how this factual inertia works. This is encapsulated in the work of Duncan Watts, a principal researcher at Microsoft Research. Watts has demonstrated, in numerous studies that explore everything from how certain songs become popular to how marketing works, that we are very good at telling stories to ourselves that sound true (e.g. Plank’s Principle). But when we subject our ‘common sense’ to the rigors of quantitative analysis, it doesn’t always pan out. So while our factual inertia is a big problem, we need to be cautious when we hear good stories about how it actually works.

Clearly, science and business, and others fields of knowledge are not abstract ventures. They’re human affairs, so they’re prone to passions and biases. Scientific discovery, in particular, occurs through hunches and chance recognition of relationships, and is enriched by spirited discussion and debate around the lab. But science is also subject to our baser instincts. Data are hoarded, scientists refuse to collaborate, and grudges can play a role in peer review. There’s a lot at play.

So new ideas take time to be accepted. And how they are accepted is far from common sense. But one thing’s for sure: Don’t write off the old folks. They have a lot to teach us.