A Look Back at Big Data Trends That Shaped 2017

There’s no doubt that 2017 was an important year for big data. From artificial intelligence to data governance, many technologies and concerns dominated the marketplace. Scott Gnau, the chief technology officer for Hortonworks, offered his insights into what big data trends emerged over the last 12 months.

AI Had an Ah-Ha Moment

Gnau compared data’s importance in the digital age to oil’s use as a raw material in the industrial age. “When I think back 25 years to the data warehouse progression, that was when the first market basket analytics were done on transactional data,” he said. “That historic data showed stores how they could refine items they sold, where to place items in a store for greater sales, and how to become more profitable in the process. That was the ah-ha moment for traditional business intelligence. We’re now having that type of ah-ha moment for AI.”

Businesses today are discovering how to use AI to create and drive new business models, which leads to better customer experiences and overall excellence in business management.

Machine Learning Created Promising Paths for Exploration

Inextricably tied to AI, machine learning presented a new frontier in 2017, making it one of the major big data trends. Machine learning itself isn’t new. But what is new are the tools available, such as Apache Spark, that now can process against massive data sets. The more data available for analysis, the more accurate the models can become, and the smarter the machines become.

Healthcare is one of the most exciting frontiers taking advantage of machine learning. For example, smart data organization Clearsense serves the healthcare sector. Their clients knew many medical challenges they faced could be solved with data, and they wanted to use machine learning. In response, Clearsense created Inception, a product that delivers real-time streaming data to help users make critical clinical, operational, and financial decisions.

Another example is IBM’s Watson for Oncology. This tool delivers evidence-based treatment options to doctors, as well as the supporting data that informs the suggestions made by Watson.

Big Data Opportunities Thrived in the Payments Industry

For the global payments industry, big data analytics provided a golden opportunity in 2017. Fintech disruption occurred on many fronts, and mobile payments and digital wallets delivered new ways people can exchange money. Blockchain technology removed the middleman from transactions and provided accurate transaction records and greater security. Legacy banking and business models were rethought and restructured.

Leveraging the big data assets that reside within this industry led to better and faster fraud detection, more opportunities for real-time payment exchange, and more sophisticated and informed customer analytics.

Data Security and Governance Became More Scrutinized

Data security and governance are evergreen concerns, but the imminent arrival of the EU’s General Data Protection Regulation (GDPR) has brought these issues to the forefront. GDPR places the control of personal data back into the hands of consumers. The regulation requires customer consent for the use of personal data, as well as the right to be “forgotten” by having their data erased. If you do business in the EU, this affects your business obligations related to consumer consent, fines for noncompliance, breach reporting, and designing infrastructure to meet privacy requirements.

Beyond GDPR considerations, as data sources become decentralized and democratized, businesses find it harder to manage governance and security restrictions. With this in mind, they’re seeking out services to help manage, govern, and secure data and workloads across sources, data types, and data stores.

The Struggle to Effectively Use Big Data Continued

Many businesses remain baffled about how to harness the power contained within their big data, but Gnau feels optimistic. “I think back to the early ’90s, when I worked in the business intelligence and data warehouse business. Industry analysts said that 90 percent of all data warehouses failed and customers remained frustrated. Those analysts are saying the same thing right now about big data,” he said. “But then I see the number of customers that show up to our big data workshops. We’ve crossed over from the point where big data is a niche science project to the point where it has demonstrable value across all industries.”

These big data trends of 2017 prove that the future remains bright. As long as businesses continue to invest in its potential, they will reap its benefits for years to come.