If you are looking for a relatively sustainable and secure job option start studying data analysis. A report by McKinsey Global Institute predicts that the U.S. needs nearly 200,000 more people with serious number crunching and analytical skills, as well as 1.5 million more data-literate workers.

Data is driving our world. Not just in business but in academia, politics, sports and even in managing our personal lives. The social web has rewarded us with an unprecedented amount of patterns to uncover, from how political ideas spread through blogs to combing Tweets to determine the level of happiness Americans enjoy in the mornings.

The data-driven self movement is gaining momentum with gadgets that perform long-term monitoring of blood pressure, sleep patterns, stress responses, even moods. With such devices we’ll have a history of our behavior and health far more accurate than our often-fallible memory.

It seems the trend will touch every area of industry. Digital sensors are already embedded in much of our equipment, cars, planes and ships. UPS and Fedex are constantly attempting to improve delivery times based on the traffic data they receive from sensors in their trucks. Police forces in Chicago and New York analyze historical crime patterns linked to weather, sporting events and even paydays, and route their forces to hot spots accordingly. Other studies have shown that increasing or decreasing numbers of real estate searches within Google more accurately predict sales than the expert real estate economists.

Welcome to the Age of Big Data. The new megarich of Silicon Valley, first at Google and now Facebook, are masters at harnessing the data of the Web — online searches, posts and messages — with Internet advertising. At the World Economic Forum last month in Davos, Switzerland, Big Data was a marquee topic. A report by the forum, “Big Data, Big Impact,” declared data a new class of economic asset, like currency or gold.

Of course this is the heart of the scientific method: Get as much data as possible and try to find a pattern, if any. Decisions will be increasingly based on data and not necessarily on gut or intuition supported by anecdotes and personal experience.

Such “data-guided” management is proving successful, according to Erik Brynjolfsson, professor of economics at the Massachusetts Institute of Technology’s Sloan School of Management. His research has shown that companies employing data to inform their business decisions were rewarded with increases in productivity of 5 to 6 percent. The study controlled for other possible causes of the increase.

The amount of data is growing at an extremely accelerated rate, increasing 50 percent per year and more than doubling every two years, according to IDC a tech research firm.

But all of this data—Twitter feeds, Web video, Google searches, various sensor data—needs to be organized and made comprehensible to computers. This is where forms of artificial intelligence, like pattern recognition and machine learning software, comes in and are gaining huge leaps forward.

And this machine learning leads to more machine learning:

Apple bought Siri in 2010, and kept feeding it more data. Now, with people supplying millions of questions, Siri is becoming an increasingly adept personal assistant, offering reminders, weather reports, restaurant suggestions and answers to an expanding universe of questions.

Of course with such a surge of personal, government and business data being shared and manipulated, privacy issues will not be insignificant. And of course there is always the human aspect of interpretation that may spoil what appears to be a fool proof, the-facts-don’t-lie way of life. Some researchers will be biased to only find data that supports their already-established theory. It will be important that data-analysis keeps an additional close watch on all potential variables affecting conclusions from huge data sets.

Christie Nicholson produces and hosts Scientific American's podcasts 60-Second Mind and 60-Second Science and is an on-air contributor for Slate, Babelgum, Scientific American, Discovery Channel and Science Channel. She has spoken at MIT/Stanford VLAB, SXSW Interactive, the National Science Foundation, the National Research Council, the S...
Full Bio