Why ‘Big Data’ Can’t Find the Missing Malaysian Plane

How can a commercial airliner vanish and remain lost for weeks in an age in which data on virtually everything is collected and maintained either publicly or secretly? The answer is “quite easily.” It is one thing to collect huge amounts of information, but it is another to be able to fuse it quickly and meaningfully.

Much has been made about the reluctance of countries involved in the search for Malaysia Airlines Flight 370 to share sensitive aircraft tracking data that could help locate the doomed plane. But that is not the only issue.

The inability to quickly connect the dots is largely a technical problem. The sensed data of the world aren’t inherently organized or co-located or structured to be interoperable, semantically or otherwise. Even if all the data were being shared openly, it is simply not easy to marry disparate, dispersed data stores and perform timely searches to find connections, coincidences or errant commercial aircraft.

This is essentially a situation in which the amount of data spread across the globe overwhelms the ability of any one authority to bring it all together and process it. How to deal with such vast amounts of data is a problem that is being examined by government agencies and others around the world.