Experiment with Organizational Change Before Going All In

Think about the last time you considered introducing a significant change in your enterprise with the intention of improving organizational effectiveness. Maybe you were contemplating a change in employment policies such as flextime or one in customer-facing processes such as a new billing system. You likely studied the proposed change in detail, discussed it at length with relevant colleagues, came up with a strategy for implementing it, and then introduced it. A crucial step is missing from this process, however: a rigorous way to determine whether the change is accomplishing its intended objectives.

Why do organizations so often introduce such new initiatives without thinking about this step? Behavioral economics, the discipline that combines the fields of psychology, judgment and decision making, and economics, offers an explanation.

When we think that a particular course of action is the correct decision, our human tendency is to interpret any available information as supportive of that course of action. This tendency is known as confirmation bias — our perspective on the world is distorted in a way that tilts toward confirming our currently held views. Furthermore, once we start down a path and invest resources in it, we tend to justify our past investments by continuing down that path even when new information suggests that the path is unwise, a phenomenon known as the escalation of commitment.

Together, confirmation bias and the escalation of commitment lead organizations to refrain from evaluating changes because the key decision makers feel (erroneously) that they already know that the changes are good ones. The unfortunate result is that organizations persist in implementing ineffective policies and fail to even contemplate the possibility of superior alternatives.

That’s where experimental testing comes in. By forcing organizations to clearly articulate their goals and then to rigorously judge their decisions by those metrics, experimental tests can help managers avoid costly mistakes and can open up the consideration of other possible solutions.

When pharmaceutical companies conduct clinical trials to test the safety and efficacy of their drugs as part of the process for obtaining regulatory approval, some patients receive the drug and others some standard existing treatment or a sugar pill (placebo). A comparison of the two sets of results tells us whether the drug improves patients’ symptoms and has side effects. Many potential changes in your organizations can be subjected to a similar experiment: implement the change in some places, but not in others, and compare performance in the two groups to learn whether the change is effective.

Why can’t you sidestep the hassle of an experiment and simply compare the performance before and after a change? In some cases, this approach is valid, but in many cases the results will be misleading. Say you introduce an innovative new customer-relationship-management (CRM) tool for your sales force, and revenue increases by 15%. The change is a success, right? Maybe or maybe not. What you have neglected to ask is what would have happened if you had not introduced the change. Revenue might have increased by 20%. Without some knowledge of what would have taken place in the absence of the change, it’s hard to evaluate your success.

A handful of organizations have already embraced the principles of behavioral economics and the experimental mind-set. One is the Walt Disney Company’s R&D department, where one of us spent a summer. After identifying areas for cost reduction or process streamlining, it would design randomized experiments to test the effectiveness of possible changes.

In one project, the group looked for ways to encourage hotel guests to reuse their towels, an environmentally friendly practice that could also save Disney money. Group members designed an experiment to test whether having guests make a specific commitment to practice sustainable behavior and giving them a pin to symbolize that commitment would lead guests to engage in more eco-friendly behavior. The result: Guests who were randomly chosen to be part of the program were over 25% more likely to reuse their towels compared to guests who were randomly chosen not to be part of the program.

Having an on-site lab may not be feasible for many organizations, but it is still possible to engage in experimental testing without devoting vast resources to the effort. Careful experiments require three key ingredients, which can often be implemented with little incremental cost.

Identify a target outcome. This outcome must be something specific and measurable. “The effectiveness of the customer service call center” is too vague and might be narrowed down to “the percentage of incoming calls successfully routed to the appropriate technician within three minutes of receipt.”

Articulate what exactly your proposed change will involve. Simpler changes are often better, since complex changes with many moving parts make it difficult to identify the component that is driving results.

Introduce the change in some places in the organization (the “treatment group”) but not in others (the “control group”). Take the unit targeted for the change and divide it into two groups. Ideally, you’ll be able to flip of a coin to determine which places are assigned to which group; randomization helps to ensure that the two groups are similar, on average, right before the experimental test begins. Any differences in outcomes between the two groups can then be attributed to the change. When such simple randomization is not feasible due to logistics, ethics, cost, or sample size, more sophisticated analytical techniques can be employed.

Testing workplace practices in this manner is important because our own intuition regarding what will or won’t work is often mistaken. Take the issue of productivity. Most of us believe that if we fall behind on deadlines or commitments at work, we should simply spend more time working. Yet it turns out that people are actually more productive when they work a bit less rather than a bit more.

One of us (Francesca) conducted a field experiment on this topic with Giada DiStefano (of HEC Paris), Brad Staats (of the University of North Carolina’s Kenan-Flagler Business School), and Gary Pisano (of Harvard Business School) at a tech-support call center in Bangalore, India. The research team studied employees in their initial weeks of training to serve a particular customer account.

They were randomly assigned to one of three groups. Each group went through the same technical training with a couple of key differences. On the sixth through the 16th days of training, workers in one group spent the last 15 minutes of each day reflecting (in writing) on the lessons they had learned that day. Employees in another group did the same but then spent an additional five minutes explaining their notes to a fellow trainee. Those assigned to a control group just kept working at the end of the day and did not write down or share any thoughts they might have about the lessons they had learned.

In a test given at the end of the month-long training program, employees in the first and second groups respectively performed 22.8% and 25% better, on average, than the control group. This was in spite of the fact that trainees in the control condition worked 15 minutes longer per day than trainees in the other two groups. Though we focused on performance during training in this study, we found that reflection had a similarly beneficial impact on how people carried out their jobs day in and day out in terms of things such as productivity in entering data, teamwork, and service quality.

Think about the most pressing questions your own organization is facing. Would employees be more productive if they had flexible hours or the ability to work from home? Would your clients be more satisfied with your services if your processes were more transparent? Would your company be more successful in retaining talent if employees had greater decision-making authority? Your intuition may suggest answers, but it may be worth it to put them to the test.

John Beshears is an assistant professor at Harvard Business School and a faculty affiliate of the Harvard Kennedy School of Government’s Behavioral Insights Group. He cochairs an HBS executive education program on applying behavioral economics to organizational problems.