I like the pace of technology, both in finance and in health care, which I occasionally stray into. I have written for Securities Industry News, Risk, Institutional Investor and Alpha. For ten years I edited Windows in Financial Services, covering Microsoft technology in finance, and I continue to write for Banking Technology and Trading Places, both in London.

Big Data -- Big Money Says It Is A Paradigm Buster

“Big data is one of the biggest transformational changes in the data center and IT landscape,” said Ping Li, a partner at the VC firm Accel Partners, which is running a $100 million Big Data fund. “It happens once in a generation,” he told the audience at a Churchill Club panel in Silicon Valley.

Gartner predicts that data will grow 800 percent over the next five years and 80 percent of the data will be unstructured.

And just what constitutes Big Data? After SC2011, the US supercomputing conference in Seattle in November, Addison Snell, an industry analyst, blurred the lines a bit in a podcast with HPCwire.

“There is small Big Data just as there is entry level high performance computing,” he said. Someone who has worked in gigabytes and now has to work in terabytes is dealing with Big Data, added the CEO of Intersect360. “It’s relative to the infrastructure you had before.” It may incorporate complex event processing, data mining and complex real-time analytics. Big Data can have many elements — large files, large volumes and real-time I/O within a short data life span. Every vendor at SC2011 was talking about big data, agreed Nicole Hemsoth, editor of Datanami.

Or to put Snell’s observation another way, Big Data breaks existing systems and ways of working.

“A lot of people know how to work with data,” observed Anand Rajaraman, “but now there is a lot more data so the kinds of things you can do with it and the way you work with it can are very different. The founder of companies which have been acquired by Amazon and Walmart, Rajaraman is now senior vice president at Walmart Global e-commerce and co-founder @WalmartLabs, and a professor at Stanford.

“The tools [for Big Data] are very different. Many of the fundamental algorithms for predictive analytics depend crucially on keeping the data in main memory with a single CPU to access it. Big Data breaks that condition. The data can’t all be in memory at the same time, so it needs to be processed in a distributed fashion. That requires a new programming model.”

This can be hard for traditional data users to understand, He watches students attack Big Data problems by creating a sample, but that defeats the value of Big Data with all its potentially informative outliers.

Businesses are catching on to the promise of Big Data said Luke Lonergan, chief technology officer at Greenplum, an analytics company that was acquired by EMC.

“Every business is looking for ways to get tighter connection with its customers, to improve prediction and and move them along a trajectory. We see a certain urgency around Big Data.”

Traditional users of Big Data — retail, telecom and intelligence — are already comfortable with it, said Lonergan. The next big set of users are in mobile-social, especially incorporating geo-location. Some areas have been underserved, such as health care, which he described as the third rail because it has been too hard and too slow. But now health care is experiencing fundamental change similar to what retail felt when customers came in armed with smartphones and had more information than sales people. Patients are starting to acquire more information and health care providers are developing more analytics.

Big Data is also going to change science, said Rajaraman.

“The way science is done has evolved over the centuries — observation, notebooks and then you come up with a theory. That was followed by algorithmic science. Now data analysis is the fourth paradigm for the new way science is being done ranging from earth science, to chemistry to biology and psychology. Science is about more and more data.”

Big Data is not an American fad — it is a global phenomenon, said Accel’s Li.

“It’s amazing how global this is. Adoption of Big Data is happening faster in some emerging markets where they don’t have to content with legacy databases.” Instead they are using open source tools with commodity servers and storage.

“I think we are in the very early days of transformation,” Li added. Accel is looking for companies working in data management and platforms.

“A lot of the applications that ride on top of these new data platforms have yet to be invented. Right now users are importing legacy business intelligence and ERP, putting it on top and calling it Big Data. I think we will see new applications that are Big Data. We are just starting to see the seeds as people recognize the potential of the new platform. We will see startups that will build new applications which will break the existing platforms and start again.”

Rajaraman agreed.

“Big Data is at a stage where early innovators are out there.” This is like the early days of e-commerce when pioneers had to figure out components like fraud and payments, he added.

“Each of those is now a big company. In Big Data, each of the new use cases will get commercialized and become a big company on its own. Data growth today is faster than Moore’s law and faster than bandwidth growth, so we need new ways of organizing data, new computing and algorithms because computational power is not keeping up.”

People underestimate the degree to which this can change their business, said Greenplum’s Lonergan.

“Those with data science teams begin to understand; others don’t see how much it can do. Rather than any single stovepipe, we are about predicting the future. Some expect they can buy an application to make Big Data happen, but it doesn’t work that way. Businesses should see an explosion of new activity around the explosion of Big Data.”

Post Your Comment

Post Your Reply

Forbes writers have the ability to call out member comments they find particularly interesting. Called-out comments are highlighted across the Forbes network. You'll be notified if your comment is called out.

Comments

Big data has put businesses in an interesting position. Not knowing how to properly collect, maintain and utilize the information can be devastating. However when properly handled, big data can catapult an organization over its competition. Fun times to be a CIO.

I’ve seen this development first hand and can agree with every word. My dissertation (back in 2004) involved the evaluation of over a quarter of a million simulated conditions. Last year in my applied work, I’ve had to manage and manipulated over a half billion lines of data. All of the issues associated with working memory, processing speed, and storage space are right on. I can only imagine some of the big data challenges this year!

Big data will change the paradigm of the industry. It will redifine mass market into micro segments. It is set to redefine the way business decisions and algorithms are designed on the front end of the business cycle. It will encompass Cloud, Mobility and digitalization assets into it. http://theignitedmind.wordpress.com/

Good article. 1 point I have to make though is that I fundamentally disagree with Anand Rajaraman’s statement in the beginning that Big Data can not all reside in memory. Memory is more and more the same price or at least competitive with disk. I’ve implemented Big Data projects entirely in memory time and time again. See www.violinmemory.com – an enabler, in my opinion, the leading enabler.

Successful startup veterans know it’s all about cutting through the VC funding clutter and finding a truly disruptive solution that solves a massive problem. Big data in and of itself is merely intriguing, but applied to B2B customer acquisition as Mintigo has done it has the potential to have a massive macroeconomic impact: