Big data bonanza

In 1945, as a long stand-off between regulated capitalism and socialist state planning was setting in, Friedrich von Hayek explained in a celebrated article why markets would ultimately triumph. The dispersed nature of knowledge, held by millions of separate individuals, meant central planners could never process enough information to allocate resources efficiently. The price mechanism, free and decentralised, gives everyone an incentive to act on what they individually know, and in turn reflects their actions. This allows markets to do the job vastly better.

Hayek was proved right by the socialist bloc’s collapse in 1989. But as the Financial Times’ “Big Data” series (ft.com/bigdata) this week shows, the ability of individuals, companies and states to collect, store and analyse information has undergone a revolution since then. Free markets will continue to outperform central planning on the whole, but no longer only for the reasons Hayek gave.

Editorial

Ever more powerful information technology now allows consumers to carry gigabytes in their pockets and businesses to organise and analyse data on a scale never seen before. People’s willingness to use the new electronic tools to communicate and share information about themselves means that even the most advanced companies are only scratching the surface of the behavioural patterns these troves of data can potentially reveal.

The greater the ability to process information centrally, the more realistic the prospect of centralised control by companies or states. Both are ramping up their capacity to monitor the behaviour of customers or citizens. Policy battles will increasingly be fought over the use and ownership of data.

Already, governments intervene to protect privacy, as with Google Streetview’s picking up of private WiFi network traffic. At the same time, fear of too much intervention just torpedoed an effort to update international telecoms regulations. Then there are those who want to exempt data from traditional legal and moral categories – from “free culture” activists and filesharers to WikiLeaks, hackers and other Che Guevaras of the internet.

Such contests will grow stronger before they are settled. Meanwhile, big data is empowering individuals, too. If the information revolution makes data more amenable to centralised analysis, it also gives individuals the knowledge better to exercise their economic freedom – price comparisons, for example. Even when companies use data to charge consumers different prices for the same good, this can enable exchanges that would be impossible at a uniform price, thereby making the market more efficient.

The horror scenario of big data enabling state or corporate power cannot be dismissed. More likely, the IT revolution is too powerful for its benefits to be monopolised.