The Mistake Companies Make When They Use Data to Plan Diversity Efforts

Executive Summary

In order to step up diversity efforts, organizations often start with people analytics to pinpoint where to intervene. But as organizations take a data-driven approach to identifying areas of change, many encounter one issue: they have a great deal of data about the experiences of certain groups, but far less on others. An organization may be able to tell a clear story about how women in general are faring, or may be able to discuss the experiences of people of color broadly, but what about Asian women compared to Black women, or Hispanic men compared to white men? With such limited data, many companies revert back to broad categories (e.g. “all women”) as they structure diversity initiatives.

But pooling “people of color” or “women” to have more data discounts within-group differences and hinders meaningful change. Research shows that “one size first all” diversity approaches often only benefit a subset of employees. And efforts directed at women broadly tend to advance white women at the expense of women of color. Research identifies four key steps companies can take to ensure they don’t fall victim to the faulty logic of small numbers: be willing to make claims based on small numbers, dig deeper, engage managers as allies, and don’t settle for small numbers.

In order to step up diversity efforts, organizations often start with people analytics to pinpoint where to intervene. But as organizations take a data-driven approach to identifying areas of change, many encounter one issue: they have a great deal of data about the experiences of certain groups, but far less on others. In working with companies seeking to improve diversity and inclusion, we have found small numbers can be a big sticking point.

An organization may be able to tell a clear story about how women in general are faring, or may be able to discuss the experiences of people of color broadly, but what about Asian women compared to Black women, or Hispanic men compared to white men? When we start to break down demographic groups, many companies struggle.

We’ve worked with organizations where the smallest groups may represent as little as 1% of the workforce and consist of fewer than 50 employees. Consider, for example, Google’s most recent diversity report: 0.5% of its U.S. technical workforce is comprised of black women and 0.9% is Latinx women. Even in a company this large, the numbers get small.

With such limited data, many companies revert back to broad categories (e.g. “all women”) as they structure diversity initiatives. In fact, articles on people analytics advise practitioners to “beware of small numbers” because it’s hard to determine what they mean.

Taking this logic too far, however, can have highly damaging effects. Pooling “people of color” or “women” to have more data discounts within-group differences and hinders meaningful change. Research shows that “one size first all” diversity approaches often only benefit a subset of employees. And efforts directed at women broadly tend to advance white women at the expense of women of color.

So, what can organizations do to better understand the experiences and outcomes of employees represented in small groups? Our research identifies four key steps companies can take to ensure they don’t fall victim to the faulty logic of small numbers.

Be willing to make claims based on small numbers. Small numbers are problematic in statistics because most analyses are based off a sample of a population. To draw conclusions about the full population, we must make inferences. Given this, we have to acknowledge that reality is found somewhere within a range of possible outcomes. With small samples, the range of possible outcomes is larger, and we have less certainty about what is truly happening.

When an organization has data on their full population, however, small samples are not an issue for identifying patterns. Understanding descriptively what’s going on with your employees doesn’t require inference from a sample to a population. Say, for instance, that of the 40 Black women in a hypothetical organization in 2017, 15% or 6 people, left the company by 2018. This is enough information to see that you might need to pay more attention to retaining black women.

Now say that the attrition rate for women on average in the organization is 6%. If the company just focuses on the aggregate, the difference in attrition rates among smaller groups gets obscured. The interpretation goes from “the turnover of black women is alarming” to “our company has a low attrition rate for women.” This can harm future retention efforts.

When you have small numbers, each loss, each hire, and each promotion matters. Small numbers can create uncertainty, but they don’t mean you can’t do analysis or take action. Instead of dismissing such trends, examine them and what they might mean about the organizational environment.

Of course, under-representation can be more severe than this. When an organization has only 1 or 2 people to draw on, it’s hard to talk about trends. In some cases, there are 0 people to analyze. A lack of people cannot mean a lack of accountability. If nothing else, an organization has discovered how much further they need to go to build a more representative workforce. 0 is a data point in itself.

Dig deeper. Descriptive work shows the ways in which organizational outcomes are patterned by race, gender, and other characteristics. In order to understand what these patterns mean and where they come from, the next step is to dig deeper. Small numbers are the perfect opportunity for gathering interview-based data.

Once an organization is aware that 15% of Black women are leaving, there are a number of questions to ask: Why this is happening? What can be done to change it? How does this affect those who remain? Interviewing people provides the opportunity to ask questions that are not picked up in the numbers. Gathering detail is crucial to provide context for a number like a 15% attrition rate, and can offer insights on potential interventions.

Interviews are also a reminder that the ‘data’ you are dealing with are people who cannot be fully captured by a statistic. Talking to employees is a chance to understand their career aspirations, specific struggles, and cultural insights. Organizations should not forget that decisions made based off the data affect people’s careers and livelihoods.

In order to make interview data most useful, its collection should be thoughtful, systematic, and flexible. Organizations need to consider what information they are seeking to guide the questions they ask. At the same time, they should be prepared to gain new and unexpected insights that inform follow-up questions.

Engage managers as allies. The above two methods provide necessary leverage to learn about small groups in an organization, but they are unlikely to have impact without the buy in and support of managers. Managers are a crucial point of intervention because they make key decisions about hiring, advancement, and the projects and teams on which people work. They shape the daily experience of employees. So they are key to advancing diversity and inclusion.

Organizations can help managers act as allies by creating an environment where crucial and often difficult conversations can take place. This starts with educating managers about intersectionality. Employees do not experience organizations based on their race or gender separately; they live their lives at the intersection of these characteristics. The organizational experience of Black women is not the sum of the experiences of Black people and women. A number of resources exist to explain intersectional approaches to diversity, why they are needed, and how to implement them. Helping managers center intersectionality can promote efforts that act on a 15% attrition rate for Black women rather than assuming a 6% attrition rate for women on average is indicative of all experiences.

Don’t settle for small numbers. Analyzing existing conditions and cultivating support is a big part of implementing inclusive diversity efforts, but the most important step for any organization is to increase small numbers. Take an honest look at your organization’s hiring, retention, advancement, and cultural practices. When you understand where the problems are, you can start to fix them.

In our research with companies, many leaders react with dismay, dismissal, and disbelief in the face of small numbers. We have long heard that small numbers are an inevitable result of talent “pipelines” beyond the control of organizations. The underrepresentation of certain groups is not, however, inherent or inevitable. It is the product of actions people take and structures people create, many of which occur within organizations. With effort, actions and structures can change. Organizations need to be part of this.

As companies collect more and more data on their workforce, there are going to be groups for whom the available data are more limited than others. Small numbers cannot be a rationale to stall progress. Concluding that little can be said with limited data renders underrepresented groups more invisible and creates a roadblock to meaningful change. To create lasting and impactful change, organizations should be willing to analyze small numbers, gather detailed interview data on employee experiences, engage managers as allies for change, and hold themselves accountable to making small numbers grow.

Katie Wullert is a Doctoral Candidate in the department of Sociology at Stanford University.

Shannon Gilmartin is a senior research scholar at the Stanford VMware Women’s Leadership Innovation Lab and an adjunct professor in Mechanical Engineering at Stanford University.

Caroline Simard is the managing director of the VMware Women’s Leadership Innovation Lab at Stanford University.