Does Big Data Mark the End of Leadership Savvy?

by Dr. Mike Hasler

Today’s CEO is faced with increasing levels of complexity that correlate strongly with the working definition of Big Data: massive Volume of date, accelerating changes in the Velocity of that data, and it comes from a growing Variety of sources. These three characteristics of data--volume, velocity, and variety--work together to create a dramatically more complex business environment. A senior decision maker may be left wondering, what is the role of my hard-earned leadership instinct and industry knowledge—must I always follow the data trends?

Data or intuition, a false choice

During the time when I was an executive in high technology in the late 90s, we often used a quip that was well known throughout the industry; “If you can’t back up your position with data, you’re just another ‘jerk’ with an opinion.” And it is this quip that really highlights the executive conundrum; do I trust my gut or the data? Put another way, what do I do when my gut disagrees with the data? We have all been in that situation: the data is collected: the analysis is performed; and the results presented. But something just doesn’t “smell” right. It doesn’t pass the “sniff” test.

This is the seminal moment; the point at which you decide whether you lead a data-driven organization, or one driven by intuition. I would challenge those facing this moment that the decision is not one of trusting the data as presented or going with your gut, but rather, it is whether you have the right data. There is a wide range of research from academia and major consulting firms that there is clear validity to intuition that is born from an individual’s experience, but the major error that leaders often commit is assuming that their own personal experience is indicative of the broad range of occurrences.

Given that most experts would agree that no model is perfect, the goal should be improved decision making, not unassailable definition of the truth. In this instance, the data that don’t pass the sniff test are best subjected to further review, refinement, and analysis. Using data and the supporting data analysis to reveal the “truths” that are buried within, leaders can reduce the complexity of strategic decision making by turning to data to enhance their intuition.

Looking under the right rocks

As organizations become more and more data-driven, managers begin to see the power and value of analysis. There can be a tendency to embrace the idea that “if some data is good, then lots of data must be great.” Unfortunately, unless the approach to data collection, storage, analysis, and implementation is understood within the context of the organizational strategy, then the results likely will not live up to expectations.

As noted earlier, leaders today have dramatically more data available due to the advent of new sensor technologies, data portals (internet applications, social media, etc.), and data storage costs that have dropped dramatically in recent years. Additionally, tools for addressing these large data sets have improved substantially.

Over that past 15 years or so, Excel has been the analytical tool of choice for the vast majority of business analysis, and many organizations have been able to use this tool to perform outstanding analysis. With software, tools, and platforms designed for larger data sets (such as SAS, SPSS, R, and Hadoop to name just a few), organizations have been able to unleash the insights from the data that were simply not possible with Excel only. Using these larger data sets with the more powerful tools can reveal powerful descriptive, historical insights into what actually happened.

In turn these same data can be used to develop predictive models to gain insight into what is likely to happen. As the data becomes more complete and the analysis more detailed, leaders can then use it to provide prescriptive insights to better determine best courses of action. The evolution from descriptive to predictive to prescriptive analysis correlates to the success of organizations to fully embrace a data-driven strategy and culture.

How fast can you respond?

The value of data is not simply in the volume, however. True strategic value occurs when leaders are able to take advantage of the data velocity to gain competitive advantage in the marketplace. The value of business analytics lies in the ability to uncover those truths and insights that aren’t readily obvious (or don’t align with conventional wisdom or intuition) to the rest of the players in the marketplace.

When the organization has the resources and the analytical capability to gather data, perform the analysis, and make adjustments to respond to changes in the market business analytics becomes a distinct competitive differentiator. The ability to rapidly implement changes before the competition in response to identifiable trends in the data defines the successful data-driven organization.

Developing data-enhanced intuition

Success in business analytics isn’t accomplished through big data and analysis alone. The best, most successful projects involved a three-pronged approach that includes subject matter expertise, information technology support (software, tools, hardware), and, of course, analytics. Once the analysis has been executed, the normal activities of effective organizational change management must be present: project management, business implementation, and clear sponsorship of the organization’s leaders.

It still holds true that if you dig deeply enough, you can likely find data to support virtually any notion. However, the real value of analytics in the business environment is not to replace managerial judgment and intuition, but to enhance that intuition with data. Having the analytical skills in your organization to recognize “truths” from the past that might contradict the conventional wisdom and take advantage of trends as they are happening will be one of the keys to differentiating your organization as a data-driven enterprise. Utilizing robust data analysis will help you avoid being “just another jerk with an opinion.”