Can Big Data Give us Big Ideas?

Over 2.5 million terabytes of data are now created every day. The pace of new data creation has reached such a clip, according to IBM, that over 90 percent of the data currently available in the world was created in just the last two years. As data has exploded, however, its uses have remained constrained to relatively small niches. Chief amongst those is targeted advertising — serving the right ads to the right consumers at the right time. And to some extent, it is easy to see why: last year, nearly 90 percent of Google’s revenue — about $90 billion — came from ads, and the story is virtually the same for most of today’s internet giants.

But, as Olivier Toubia points out, as computers become more adept at processing unstructured data — like photographs, video, and text — Big Data holds promise in an increasing range of areas. And according to Toubia, it may even help us be more creative.

“You can think of an idea as a recipe,” Toubia, an expert in innovation who has researched idea generation with fellow Columbia Business School professor Oded Netzer, explains. “You combine pre-existing ingredients and turn them into something completely new.” The trick, of course, is getting the balance right. Nowhere is that more true than when it comes to creativity, where, as Toubia points out, the ideas perceived as the best or most creative involve a careful balance of the familiar and the novel.

This effect has been well documented across a range of areas, from marketing, where consumers are shown to prefer product names that remind them of parent brands without being too close, to scientific papers, where papers that cite a common mix of journals but add to them a few unexpected ones tend to have the highest impact. “You want some similarity, some analogy,” Toubia says. “But you also need a twist, a breakthrough, something new.” In other words: stray too far from existing concepts, and your idea is likely to be seen as half-baked; stick too close, and you’re unlikely to generate much excitement.

Efforts thus far to engineer creative machines have largely fallen flat — as Toubia reminds us, creativity “requires intuition, connecting the dots, seeing the big picture, and taking novel perspectives,” all weak spots for computers. Big Data, however, may be able to help in a more prosaic way. “The first step in idea generation — coming up with a list of ingredients — is really just information retrieval,” an area where computers, with their vast memory and speed of recall, excel.

“If an idea is too familiar, Big Data could suggest ingredients that would make it more novel. And if the idea is too novel, it could suggest ways to make it more familiar.”

As a first step, then, existing technologies could be used simply to help innovators develop that initial set of ingredients. But, as techniques like text mining and semantic network analysis mature, data could be leveraged to evaluate an idea’s creativity and aid users in refining it. “If an idea is too familiar, Big Data could suggest ingredients that would make it more novel,” he says. “And if the idea is too novel, it could suggest ways to make it more familiar.”

Such an application of Big Data would have benefits for entrepreneurs and multinational corporations in almost any industry, Toubia says. But given the breadth of information available, expanding the use of data could have benefits for all of us — from helping individuals learn more efficiently, to better organizing our time, to helping us make better choices about products and services. Such interventions are not merely good for consumers. By fostering trust, they can spur loyalty that’s essential to companies’ bottom lines. “There’s a huge waste of data out there,” Toubia says. “There’s so much more that firms could do to help peoples’ lives.”

About the researcher

Olivier Toubia

Olivier Toubia is the Glaubinger Professor of Business at Columbia Business School, where he also serves as Faculty Director for the Lang Entrepreneurship Center...