For those wondering about the big picture and some of the nitty-gritty details (metadata, data visualization, open document formats) The Economist has a killer package on big data. Download it at the web site, or wander out and get your own copy of the magazine. I was intrigued by the idea that the “it” career in a data driven world was statistician.

Advertisement

Writing about the concept of big data is kind of like trying to write about water. Water is essential, touches so many aspects of life — from evolution to the current location of cities — that one article, one book or even one field of study can’t articulate its influences. Data will be the same way in the not-t00-distant future, thanks to cheap, scalable computing and ubiquitous broadband enabling a connected everything.

The explosion of data is affecting every aspect of our daily lives and how we make decisions, so itâ€™s no surprise that statistician is the â€œitâ€ career in our information-driven world. I would go a step further and suggest that even more important is the need for data analytics experts who can master the art and science of using their data warehouse and business intelligence systems to help their organization make critical decisions. The current workforce is not prepared for the big data decade we are facing, and there is a real need for both the business and IT communities to become more focused on analytics. That’s because even though data may be ubiquitous like water, finding the “hidden gems” isn’t easy, and companies big and small need to truly understand the “why” of their data (See this Teradata magazine article – http://www.teradata.com/tdmo/v08n01/Features/CoverStory/HiddenGems.aspx).

Our company is responding to this rising demand for specialists who can make sense of vast quantities of data through initiatives like Teradata University, a web-based portal for faculty and students in data warehousing and BI that is provided at no cost to universities. In December, we also hosted a BI congress with business leaders, IT experts, and college professors in an effort to bridge practice and academia by exploring at how to teach students in the industry and where our research should focus. These types of partnerships will help us all begin to catch up with the proliferation of data and more fully realize the benefits of data as a resource.

Yup, statistics will be the “it” career in a data driven world. Hal Varian, Google’s Chief Economist calls it the “sexy” job of the upcoming decade (http://www.youtube.com/watch?v=D4FQsYTbLoI)
The challenges with data and its massive growth have been around for a while now. On an average statisticians spend up to 80% of their time in the trenches cleaning and integrating data. For most individuals that is not an aspirational career.
However, with the emergence of BigData as a formal discipline and accompanying technological innovation, we can now provide statisticians with the tools that allow them to analyze massive amounts of data very quickly without the mundane chores. This gives them the unique capability to predict outcomes in a timely manner and potentially steer the course of a business – now thats really “sexy”

Krishnan, I think you’ve touched upon something there. The emergence of BigData is great, but it’s the tools one uses to analyze the data to draw from it new stats, facts, trends. BigData creates just a much bigger picture than we once had, but is only as good as your tools!

Data has always been there but we have had to look for it at different places [libraries, archives, government offices, books, journals e.t.c.] which was a time consuming exercise. All that has happened now is that all the data are now being transferred to a central location called the web which we can shift and search through the medium called Google.