Sramana Mitra: One thing that excites me about big data is the use of learning technologies. Is that part of your equation?

Michelle Chambers: Yes, machine learning is part of the equation, and typically people are considering a lot more factors in their analyses. What is exciting about it is not how the data is organized – the number of observations – but it is the number of variables included in the analysis. The ability to rapidly deploy that is a game-changing piece for the organization.

SM: Where is the industry going? Where do you see white spaces? Michelle, you are looking at product[development], so I am sure you do a lot of white space analysis. Please give us some color on where you see open problems that deserve to be solved, or if they are solved will have a significant impact.

MC: It is clear that the industry is going to the next generation of analytic applications. Those are going to be deployed not just on a traditional B2B or B2C model, but on a convergence of crowd sourcing to provide some of that. You are going to see models that evolve which are going to be B2B-C and C2B-B kinds of models in the marketplace.

I think it is beyond open source. I think open source is the key element to the foundation, but to then the next step is to take domain expertise and find a way to deliver it in [a way that represents] business value that enterprises can leverage across the board. You are going to see that in the next few years. You will see it combined with cloud deployment around analytics and much more platform as a service and software as a service analytics. You will also see many more analytics applications that are on a consumer basis. You will start seeing that emerge around personal healthcare.

You will also see the deployment and the need for more real-time analytics, and also what I call close loop analytics. The real time is going to force a close-look type of analytics. As your data is streaming by, you have a model that evaluates and does complex event processing based on the learning as the model is adapting. That data is [changing], and if you have real-time data streaming, what you are going to see are many more of today’s Hadoop being used as a catchall for that, or other large data platforms being used to capture that information, then for people to reevaluate their models on an ongoing basis and push them back out into the real-time stream.

I think you are going to see this next generation moving much more toward consumerism analytics. But you are also going to see analytics becoming more pervasive in terms of business operational processes. You are looking for innovative use cases, and most people today are doing the same old things. You go from industry to industry or from business to business, and all are copying each other. But there are ways to look at the value chain, look at it from the consumer perspective, and find ways to apply analytics that drive top-line revenue and do it very cost efficiently. What is going to happen next is you are going to see this entire generation of applications that people are going to use to generate top-line revenue for their business, but they will do it much more cost effectively than they could with human resources. That is one of the big trends.

SM: I have been talking with a lot of people. I will give you a quick summary of what I am hearing or seeing. Of course, there are very big data–intensive areas where there is a lot of activity. These deserve the kind of attention they are getting, and there is a lot of money to be earned there. A classic example is customer analytics. There is a ton going on in getting increasingly accurate in targeting customers in marketing functions. That is a gigantic area, and people are going at it from various perspectives. There are some big buckets like that.

Then I am seeing interesting niche applications of big data. You take big data principles and apply them to niche problems. I have seen very interesting case studies such as energy traders using big data analytics to predict energy prices based on data. Those are applications that have specific users or target customers who have the need for that kind of analytics. I have even seen some applications like this in the area of recruitment. If you follow our blog’s big data series, you will see a lot of these stories covered in detail – all these different flavors of applications. The other thing that is coming up is the presentation layer. I hear there are gaps there. The data visualization layer is one – people who design big data applications are not very happy with what is out there in the presentation layer.