On Algorithm Wars and Predictive Apps

Isaac Lopez

Predictive apps are going to be one the next major disruptions in technology as enterprises begin to take advantage of the combined power of big data and predictive analytics says Forrester analyst Mike Gualtieri.

“When we ask companies what is most important to them now, you know what it was three years ago: do more with less. Now you know what their number one thing is: innovation and customer experience,” explains Gualtieri explaining a little about the driving force that will fuel the trend.

The Forrester analyst, who describes himself as a former “failed data scientist,” spoke at the SVForum’s Big Data Analytics Conference last month about trends that Forrester sees emerging through the Big Data din. There are four major trends emerging, said Gualtieri, all of which appear to orbit around the customer sun, as companies try to use their resources to be more competitive, and attract more dollars.

Of course, any big data discussion isn’t complete if it first doesn’t start with a definition and Gualtieri offered no exception. According to Forrester, the Holy Grail definition of big data is that it’s the “frontier of a firm’s ability to store, process, and access (SPA) all the data it needs to operate effectively, make decisions, reduce risks, and serve customers.”

Forrester’s SPA approach purposefully runs against the grain of the popular “three V’s” that have taken root in the big data dialogue says Gualtieri, who believes that the idea that linking storing, processing and accessing data and then connecting those processes to business objectives is a superior way to define the phenomena. “The three V’s, volume, velocity, variety – those are measures of big data – very interesting, but not a definition.”

Big Data Means All Your Data

Getting the definition established, Gualtieri noted that the first trend that he’s seeing emerge is that big data increasingly means all of the enterprises data – and apparently, few enterprises are quite sure how to get that accomplished.

“It’s not just social data; it’s not just data in Hadoop; it’s all the firm’s data,” commented Gualtieri noting that one of the key questions that enterprises are asking right now is how to break down the silos and bring their data together in one central place where it can all be accessed. “They don’t want to just look at Omniture data on their weblogs and have CRM data over here, and have transactional data over there. They want it together.”

The approach seems to be a sort of “shoot first, ask questions later,” only instead of shoot first, it’s “gather data now.” Gualtieri said that in a recent survey of business intelligence professionals, they said that they are only using 12% of their current data for analytics (which is a common refrain echoed in the big data conference halls). Nevertheless, says Gualtieri, enterprises are undaunted – they want all their data in one place; an entire data universe at their disposal. This includes the increasing amount of public data sources that are coming online every day. One of the top questions that Gualtieri says enterprises ask about is how to pull in all these external data sources.“

However, Gualtieri noted with disappointment that when they asked people what their key inhibitor is, managing and integrating data from a variety of sources was the top response. “That’s one of the key challenges of Hadoop, too – getting all that data together and mediating it; doing ETL and processing it – quality as well,” he explained. “This is still a very key issue and a key challenge.”

The second trend that Gualtieri noted was a definite shift from “wow, there are massive amounts of data out there,” to “what the heck can we do with all this data – what can we find in it.” This, he notes, is all about the shift towards predictive analytics. “I call it the ‘Algorithm Wars,” he added.

In our discussion last week with Gartner analyst, Henry Morris, he revealed that while VC’s have previously been investing in the underlying infrastructure and data management part of the big data stack, VC’s were starting to talk openly about investment happening up the stack, where software and analysis tools (I.e. machine learning) drive value from the data.

Gualtieri seems to agree, at least in principle. “I call them robot armies, because the algorithms are the things that scour through the data and do all the work,” he said explaining that businesses are starting to wise up to this fact and hold their algorithms close.

Algorithms can be the secret sauce for an enterprise, said Gualtieri, noting a company called BloomReach which offers retailers like Williams-Sonoma and Neiman Marcus the means to take their traffic and turn it into predictive insights (such as the persons intent) based on what they know about the person – information like how they came to the site, what search terms they used, etc. They then use this information to help convert them by predicting what content they will be most interested in and building a landing pages geared towards that profile. “We see a lot of companies popping up like that which have a very algorithmic secret sauce that they’ll use.”

The democratization of data science is another big trend happening in big data, says Gualtieri, noting that data scientists and business people have vastly different skill sets.

In the data discovery lifecycle, Gualtieri explains that data scientists often tell him that they spend 70% of their time on the front end of the data – putting it all together, prepping it, and making it usable. “I talked to a data scientist at one of the Blue Cross Blue Shields – a master in statistics – he told me that he wrote hundreds of lines of SQL taking data out of data warehouses and other sources just to get the data ready to run, and then he was trying to create a predictive model to prevent re-hospitalization.”

This part is a huge iterative process, because the data scientist has to do the data prep, then figure out the modeling phase and which algorithms to run against the data. This turns the data scientist into a bottleneck, driving the trend towards the democratization of data science towards the business people and making it easier for them to access the algorithms and array of tools that turn data into insights.

The key, says Gualtieri, is compressing the data processing lifecycle for both data scientists and business people and there are a number of companies that are working to fill in these gaps. He noted that what most of these vendors are doing is creating a new set of tools that will let business people iterate and explore by choosing from an established menu of algorithms, or visualizations where they can experiment with the data and see what comes out on the other end.

“What some companies are doing, because they have the compute power – they’re just saying run them all and show me the list,” said Gualtieri who noted that this notion of auto-discovery might actually be easier in some cases than traditional business intelligence.

While Gualtieri says that Forrester believes that thinking predictive in all business processes and customer experience will become a norm, IT will have to loosen up on data governance in order to enable these trends. “A data scientist doesn’t go to you and say ‘I need these 10 fields’ or ‘I need these hundred data elements.’ The data scientist says ‘I don’t know what the heck I need, give it all to me,’ and IT is not used to hearing that – they’ve got to wise up to that.”

Gualtieri importantly notes that democratization of data science does not mean throwing out your data scientist with the bathwater. He notes that it’s about having the right tool for the job, and that like with most successful technical projects, it’s about having an ensemble model addressing particular aspects of the challenge.

Predictive Apps as the Next Big Disruption

Predictive apps are the next big innovation, says Gualtieri, describing the mashing of predictive models into the apps that people use in the embedded devices in their lives. As noted previously, Gualtieri notes that a clear shift has been made in what companies see as important priorities – where previously doing more with less was the top priority, a shift has happened where innovation and the customer experience has taken enterprises top priorities.

There are four distinguishing principles between a regular app, and a predictive app, says Gualtieri, the first of them being where big data is at most concern – knowing your customer. Where previously marketing was about personas and segmentation, the next era of software design will be customized to the individual, giving each a personalized experience. An example is Pandora’s music service, which learns about an individual’s music tastes and then adjusts what music it delivers to that person.

The next principle of a predictive app, is detecting “in the moment” intent. Gualtieri points to Google Now as a great example of how intent detection is finding its way into applications. “Google Now – at first I thought it was creepy, but now I like the whole notion,” he explains. “I take the bus to work, and [through the course of use], all of a sudden Google Now put up a card with the bus schedule. Whoa – it watched me, it followed me, and then it said, ‘Mike you take the bus, I’m going to help you out.’”

As the app recognizes the individuals “in the moment” intent, the next principle of a predictive app, he says, is morphing the content to match your detected intent. Gualtieri says we’re not too far off from a point where our devices are talking together, and conspiring to deliver us customized experiences, such as restaurant menus on a tablet customized to our preferences. Is the individual out for a jog? They stopped into a restaurant, can they eat the cheesecake?

The final piece to this trend, says Gualtieri, is that predictive apps are optimized for the device that the individual is using, completing the customization of the experience.

“Businesses can use this whole notion of a predictive app to become disruptive in their competitive environment,” he concludes, “and what this means is they have to intensify and individualize the user experience.” In order to accomplish this, he says businesses will need to hire three additional people: a psychologist to understand the users and individualize for them; a data scientist to create the predictive models; and a personal user experience designer who can take the information and design the user experience.

Let the algorithm wars begin.

Isaac Lopez is the managing editor for Datanami. Follow him on Twitter at @_IsaacLopez.