This is the first of a series of three pieces from the inaugural issue of Nielsen’s Perspectives on Retail Technology, a publication of Nielsen’s Buy Technology group.

Data analysis was historically the preserve of specialists, aided by an IT organization responsible for the enterprise’s data warehouse, business intelligence and master data management platforms.

This approach delivered enterprise-class business intelligence, but too often required companies to sacrifice flexibility, responsiveness and autonomy in return. Heavyweight business intelligence tools turned IT and data analysts from enablers into gatekeepers, holding business people at arms-length from their own data.

In an era of business stability, there was no incentive to innovate a solution to this problem: things worked well enough to be acceptable and enterprise business intelligence delivered good results. Then the digital revolution tore up that accommodation. Mobile technology and the Internet of Things allow business processes to be minutely monitored in real time. Almost every employee in a digital business can benefit from having the tools needed to turn data into insight. Enterprises that fail to democratize data analysis quickly fall behind their competitors.

How did business intelligence break free of its centralized command-and-control heritage to meet the challenge of the agile, democratic, digital world?

THE RISE OF PEOPLE POWER

Before 2001, serious software was designed for business use by organizations, but the explosion of personal-use computers, the internet and mobile devices caused developers to pivot and begin designing software for the individual end-user or consumer, rather than the organization. Software became more personal and usable as a result.

The process of consumerization was accompanied by the rise of open-source software. Open source made business-grade software available at no cost and let tech-savvy users experiment without having to go through the gatekeepers in IT and finance. A decade after the first emergence of open source in business, users were further aided by cloud vendors, who provided environments into which they could deploy open-source software—or in which it was already deployed—cutting them free of the corporate infrastructure.

At the same time as the cloud was taking off, a generation of “digital natives”—people who had grown up with computers as an ordinary part of their lives—began to enter the work place. They used consumerized software and did not see programming as the preserve of IT specialists, but as a widely useful skill: programming wasn’t just for science and engineering, but the social sciences and humanities as well. Digital natives expect to be able to use software to solve their problems, and have the chops to do so.

WITH ONE BOUND, DATA WAS FREE

In the late 2000s, consumerized data discovery tools and digital natives came together to light a fire under traditional business intelligence. Tools such as Qlik, Spotfire and Tableau could run on a user’s own computer and could ingest, process, explore and visualize data. These self-service, all-in-one tools let users sidestep their IT department’s heavyweight business intelligence toolkit and long lead times.

Unsurprisingly, data discovery tools experienced huge and rapid growth, and quickly became as important in the marketplace as the established business intelligence platforms of enterprise vendors.

Data analysis had been democratized.

NEW BARRIERS: BIG DATA AND ADVANCED ANALYTICS

Just when it looked as if a new paradigm had overtaken business intelligence, the marketplace was sideswiped by big data and advanced analytics.

The rise of big data shifted the center of gravity of business intelligence away from the enterprise data warehouse and corporate spreadsheets. Big data required that analyses include dozens of new data sources: website and system log files, sentiment and advertising data, measurements from the Internet of Things, partner data, weather data, econometrics and more. Both traditional business intelligence tools and data discovery tools were born in the world of the corporate spreadsheet, data mart and data warehouse—not big data. Vendors quickly gave their tools the ability to connect to big data, usually through SQL adaptors to Hadoop, but the experience was not that of a tool designed from the ground up to tackle the challenges of the digital environment.

At the same time, the digital transformation of business created the need for enterprises to identify and exploit “business moments.” Business moments are transient opportunities for consumers, businesses and machines to interact. For example, an internet-enabled washing machine could sense that it was about to run out of detergent, initiating an interaction with the washing machine’s owner and a set of approved suppliers to replenish its supply.

Digital business puts a premium on advanced analytics: the ability to predict, simulate and optimize outcomes. Business intelligence and data discovery tools excel at answering retrospective questions—such as “what happened?” and “why?”—but they don’t have a track record of addressing predictive questions.

A number of tools were developed or repurposed to support advanced analytics on big data. These included open-source software (R, Python, SPARK) and proprietary tools (most notably, offerings from IBM and SAS). The majority of advanced analytic tools are workbenches designed to allow a tech-savvy data analyst to build their own discovery and predictive applications. In other words, these tools took data analysis away from the business user and returned it to the IT department and the quants.

DEMOCRACY IS COMING (BACK)

Despite these limitations, the benefits that self-service data discovery tools brought to business intelligence were far too great for companies to be willing to give them up. As a result, traditional enterprise business intelligence vendors quickly built data-discovery capabilities into their products, while discovery vendors began to add enterprise features, such as sophisticated security, to their tools. Business intelligence is converging on the self-service data-discovery paradigm, but implemented on a platform that provides enterprise class security, scalability, performance and management.

Big data is also firmly established as the future of business intelligence. Vendors are enhancing their products to run data discovery against larger and more diverse datasets and support complex analyses. This is work in progress, but mature offerings are in the pipeline; the technical challenge is one that has been faced and solved several times before.

Advanced analytics is a tougher nut to crack—it is the realm of data scientists and quants, and requires a level of knowledge and skill far beyond the tech-savvy user with another job to do. Most companies are only able to meet their most pressing needs for advanced analytics, because the demand for data scientists far outstrips supply. This pain point is a huge opportunity for any vendor able to democratize advanced analytics, even to a limited extent. As a result, the problem is under attack from a number of converging directions.

Hard-core data scientists would benefit from being able to use the visual data-discovery capabilities of business intelligence tools on big data as part of their advanced analytic workflow. Business intelligence vendors are working hard to enable that.

Vendors are also looking to help non-expert users by creating consumerized advanced-analytic tools that enable them to carry out pre-defined or simplified analytics for themselves. The logic of democratization is obvious: data-literate users are in far more plentiful supply than data scientists, and with the right tools could help companies plug their data-science skills gap.

Looking further into the future, a great deal of work is being done to bring machine learning to bear on advanced analytics. University researchers have already demonstrated data science machine assistants that are able to generate automatically the best predictive models for a dataset and algorithms that mine big data for insights; these data science assistants are closing in on parity with an average data scientist. Once they are sufficiently mature to be useful, these capabilities will be implemented in commercial and open-source tools.

CONCLUSION

The trajectory of the democratization of data analysis is clear: Self-service data discovery will expand to encompass both traditional data sources and big data, and will then expand further to encompass advanced analytics.

There is a desperate and increasing need to enable non-specialist users with full-time jobs to undertake self-service data exploration and analytics on big data.

It’s too early to say which of the candidate solutions will be successful. But what is certain is that, with the amount of data on earth set to grow ten-fold by 2020, according to the IDC’s 2014 Annual Digital Universe Study, the democratization of data analysis is an unstoppable force in digital business.