Let Data Ask Questions, Not Just Answer Them

The bigger the data, the more profitable and productive predictive analytics can be. But that’s conventional wisdom. Innovators more intent on inventing the future than predicting it should look hard at how cutting-edge scientists now computationally massage their big data. “AI” — artificial intelligence — is giving way to “AH” — automated hypothesis. AH, not AI, will increasingly inspire tomorrow’s breakthrough innovation.

As The Economist recently observed: ”More than 90 groups of scientists are now developing hypothesis-generation software. They hope to use it not on recipe books but on the vast corpus of scientific literature (by one tally at least 50m scientific papers) that has piled up in public databases.” In other words, data-driven scientists worldwide recognize that petabytes and exabytes can make computation as creative and imaginative as imagination for hypothesis generation. They’re investing accordingly.

The ingenuity is compelling: instead of using data to solve problems, AH technologies generate portfolios of provocative problems to solve. Or, more accurately, hypotheses worth testing. Global enterprises and entrepreneurs alike will be able to use big datasets to generate business hypotheses around innovation and new value creation. Instead of recommendation engines that suggest what book to read or TV series to watch, AH engines will propose new titles and concepts for media consumption. Does anyone doubt that Amazon and Netflix want to use their exabytes of data to craft new products and services for customers and partners?

Similarly, you can be sure that many — if not most — highly-secretive quantitative hedge funds have highly proprietary AH engines relentlessly recommending trading strategies and investment themes that lend themselves to review and refinement. Yes, high-frequency “algorithmic” trading of Flash Boys fame is impressive, but its investment victories are executional. The global interoperability of big datasets guarantees that successfully managing a portfolio of investment hypotheses has become an essential precursor to successfully managing an investment portfolio.

Designing systems that ask the right questions can prove more valuable to aspiring innovators than those simply giving the right answers. The key insight from my recent book The Innovator’s Hypothesis: How Cheap Experiments Are Worth More Than Good Ideas is that the ability to run fast, frugal, and scalable experiments based on high-value business hypotheses is becoming a new core competence for innovation success. As companies gather more data about their customers, channels, usage, complaints, social media, etc., we won’t just see people analyzing data with optimization in mind; we’ll be seeing machines generating “innovation hypotheses” recommending new configurations, bundles, features, pricing schemes, and business models to test. The breakthrough innovator’s hypothesis doesn’t have to come from a human. The odds are, it won’t.

High-impact innovators will increasingly rely upon AH-aided epiphanies and insights to trigger their creativity and innovation prowess. Instead of scouring the data for interesting patterns, the innovation challenge will be determining what hypotheses are worth the most immediate and innovative experiments that can scale into a valuable new product, new service, new process, or new UX. The innovation collaboration between AH harvesters and real-world experimenters will increasingly shape corporate culture.

Mash-ups of automated hypotheses and predictive analytics will be inevitable. Automating hypotheses that will excite marketers and financiers as well as engineers and designers will be as much a computational art as a software science. It’s like having a microscope, telescope, or MRI that can say, “This looks interesting. What if we tried this…?”

Of course, things get even more interesting and innovative when we start mashing up AH with “ML” — machine learning. But that’s just a business hypothesis.