Do Your Data Scientists Know the ‘Why’ Behind Their Work?

Executive Summary

Data science has been around for a long time. But the failure rates of big data projects and AI projects remain disturbingly high. And despite the hype, companies have yet to cite the contributions of data science to their bottom lines.

Why is this the case? In many companies, data scientists are not engaging in enough of softer, but more difficult, work, including gaining a deep understanding of business problems; building the trust of decision makers; explaining results in simple, powerful ways; and working patiently to address concerns among those impacted.

Managers must do four things to get more from their data science programs? First, clarify your business objectives and measure progress toward them. Second, hire data scientists best suited to the problems you face and immerse them in the day-in, day-out work of your organization. Third, demand that data scientists take end-to-end accountability for their work. Finally, insist that data scientists teach others, both inside their departments and across the company.

Recently, Ron Kenett, the distinguished Israel-based data scientist, and I compared notes on our own successes and failures — and those of our colleagues — in helping companies with data science. It was immediately clear that the biggest successes stemmed not simply from technical excellence but from softer factors such as a deep understanding of business problems; building the trust of decision makers; explaining results in simple, powerful ways; and working patiently to address dozens of concerns among those impacted. Conversely, otherwise excellent technical work died on the vine when we failed to connect with the right people, at the right times, or in the right ways.

In many companies, data scientists are not engaging in enough of this softer, but more difficult, work. Two underlying reasons contribute. First, many data scientists are much more interested in pursuing their crafts — namely, finding interesting nuggets buried in data — than they are in solving business problems. In some respects, this is natural. After all, they are taught a narrow focus on data and the tools needed to explore it, and doing so helps them earn peer recognition. Plus, applying advanced techniques is more fun than dealing with the messy realities of corporate life.

The second reason: From the company’s perspective, the talent is rare and protecting data scientists from the chaos of everyday work just makes sense. But doing so increases the distance between data scientists and the company’s most important problems and opportunities. Exacerbating this, for many organizations, data scientists are new and unfamiliar, and companies are still learning how to manage them. It is tempting to bolt data science onto your existing organization and hope for the best.

So, what should managers do to get more from their data science programs?

First, clarify your business objectives and measure progress toward them. While data science does require initial investment, you should expect real results — in terms of cost savings, new revenue, improved customer satisfaction, or risk reduction — within a couple of years. Obvious as this sounds, the implications are profound. For most, it means recognizing that you are not ready for overhyped technologies, such as machine learning, and focusing first on more basic opportunities, such as putting operational processes under control, improving data quality, and developing a deeper understanding of customers.

Second, hire data scientists best suited to the problems you face and immerse them in the day-in, day-out work of your organization. Technical competence matters, of course. But you should also seek to hire those who are curious about your business and are passionate about helping you make it better. Then make sure they are fully connected to important stakeholders and the rough and tumble of work. Beware of creating silos of data scientists. Instead, consider embedding them in the departments they support.

Third, demand that data scientists take end-to-end accountability for their work. I cannot overstress the importance of pre-analysis work, particularly understanding the problem. Without a clearly articulated problem statement, the ensuing work is just a fishing expedition. Sorting out that problem statement can be complicated by competing agendas, fear, and muddled thinking on the parts of those who own the issue. It takes skill and patience, especially for new data scientists, who are eager to show what they can do. Veterans know better. A clearly stated problem can cut through the political haze, and it suggests simpler, more powerful solutions that may not even require data science. In my work as an adviser with companies, I often find that over half the value I add is in helping them understand their real issues!

Post-analysis work is similarly critical, as insights and algorithms must stand up to the rigors of the real world. Of greatest concern are seemingly minor political issues, again testing the patience of inexperienced junior data scientists. More senior data scientists know politics can also work in their favor, and they make time to engage all impacted by their work.

Finally, insist that data scientists teach others, both inside their departments and across the company. Everyone benefits when they use a bit more data science in their jobs, but most people don’t have the needed skills. Well-placed training and a little encouragement can go a long way in helping people complete simple projects. Your data scientists are uniquely positioned to deliver that training and coach people along. This will also help data scientists learn the business.

Trust is a common thread throughout these recommendations, and managers must insist that data scientists work to earn that trust. And they must give them a fair chance to do so.

To illustrate these four points, consider a data scientist, who was employed by a new division inside a high-tech company and tasked with capacity planning for his division’s network (I served as his informal adviser). Network planning is notoriously complex. A network that works “almost all the time” means delays during times of peak demand, which angers customers, threatens service level agreements, and damages reputations. But the costs of improving performance during peak times can grow at a frightful pace. So, it is essential that business leaders understand the trade-offs.

This data scientist introduced the trade-offs to senior leaders of the division this way: “First we have to decide what sort of network we want. Loosely, we can have a ‘papa bear network,’ a ‘mama bear network,’ or a ‘baby bear network.’ And roughly here are the implications of each.” His analogy helped ground decision makers, got them thinking about business implications in new ways, and helped them understand why they needed to understand “cost vs. availability” graphs. He immediately earned a measure of trust from senior leaders. In turn, as a full participant in the discussions, he developed a deeper appreciation of the company, its long-term plan, desired market position, and values.

It is easy to see the initiative taken by this data scientist. Indeed, I recommend that others follow his lead. Note, too, that this division hired him as much for his ability to work with others, his eagerness to learn the culture, and his willingness to fully engage on the tough problems as much they did for his technical abilities. And that his bosses helped him do so.

Data science is a team sport, but it’s not a game. Managers must make clear that the goal is to improve the business, and they must hire those that can help them do so. They must do all they can to integrate data scientists into their teams and they must insist that data scientists contribute in every way possible — before, during, and after the technical work.

Thomas C. Redman, “the Data Doc,” is President of Data Quality Solutions. He helps companies and people, including start-ups, multinationals, executives, and leaders at all levels, chart their courses to data-driven futures. He places special emphasis on quality, analytics, and organizational capabilities.