Firms Won’t Experiment

I’ve often tried to help [convince] companies do experiments, and usually I fail spectacularly. … Companies pay amazing amounts of money to get answers from consultants with overdeveloped confidence in their own intuition. Managers rely on focus groups — a dozen people riffing on something they know little about — to set strategies. And yet, companies won’t experiment to find evidence of the right way forward. I think this irrational behavior stems from two sources. One, … experiments require short-term losses for long-term gains. … Second, there’s the false sense of security that heeding experts provides.

More here (HT Tyler). Wow – and I was puzzled by firms disinterest in prediction markets; experiments have a much better intellectual pedigree. Hiring consultants allows one to affiliate with prestigious folks, while focus groups allow one to brag about “listening to the people” (which is in fact how prediction markets are usually sold). Apparently actually improving decision quality is way way down in the manager priority list.

It’s likely that managers don’t so much have a problem with experimenting as with doing so openly. Not (only) because of the message this sends to outsiders that they don’t really know the right path ahead, but more importantly because it opens everything they hand down the chain of command to question. Companies I’ve worked for have tried internal initiatives that didn’t work (e.g. incentive schemes, projects, employee regulations) and then tried alternatives, but each alternative was presented with a semi-plausible story about positive goals to sell the new initiative. I’m sure managers often say to one another things like “well let’s try this out for 6 months and see how it goes”, but once the decision is made it is invariably presented to everyone else as a strong ongoing commitment. I suppose that at the highest-profile levels (C*Os and boards of large public companies) it becomes more difficult to conspire to experiment.

jsalvatier

I think there is more to this than “firms won’t experiment”. I studied chemical engineering in school (recently graduated) and worked at two different paper manufacturing firms as an intern. My projects at both firms involved conducting and evaluating experiments, and I observed others do several of their own (expensive to run) experiments. These were well organized and relatively formal experiments.

Many school projects designed to look like “real world projects” were also formulated as experiments done within a firm, so I suspect experiments are widespread.

Perhaps firms that are not used to conducting experiments are reluctant to start? Or perhaps people are reluctant to do experiments on people?

And the consultants aren’t interested in using prediction markets since they are valued for confidence not accuracy?

david

Wait. If private firms owned by self-interested shareholders won’t invest in a method that’s supported by self-interested betting… surely there’s an element of contradiction here? Why are we assuming that shareholders are failing to be self-interested but the predictors will not be? Are principle-agent problems this large?

And, well, prediction markets are driven by people who think they know the future. Firms are run by people who think they know the future. Ultimately someone has to be doing the predicting, and if they are, why not just buy into the relevant firms directly rather than by betting in a prediction market? The returns there can hardly be greater.

We should observe relatively little experimentation within firms if markets really do punish mistakes quickly. If they don’t, then… well, depending on the type of market failure here, prediction markets might not do so well either.

J Bell

i would just like point out that you are using the fact that one economics professor is anecdotally saying that firms don’t experiment to say that firms don’t experiment.

just because ariely has found that a few companies he has worked with might not do experiments doesn’t mean that all or even most companies don’t do experiments…maybe he affiliates with the wrong companies

now believe me, i do agree with what he is saying

working in a big firm, my intuition is that in general firms don’t like to experiment…but how can you use the intuition of a prestigious person you are affiliating with to point out the folly of firms doing the very same thing?

Most web companies (like Amazon and Google) do hundreds of experiments per year, mostly in the form of “split testing” (or “a/b tests”). I’m sure that large brick-and-mortar retailers like Wal-Mart do the same.

Companies which use to market to experiment flourish. Apple took a gamble with the iPod division, which more than made up for the failure of the Newton. Google has hundreds of products in beta, some of which will bear great reward, many of which will languish. Toyota experimented with introducing a fuel efficient car, propelling it to the title of world’s largest automaker. As discussed in a interview for [em]Econtalk[/em], Wal-Mart gives even low level employees great leeway to try discounts and promotions, with cash incentive.
The lack of experimentation is due to two factors. Bad managers suggest bad ideas for experiments, which would fail, directly revealing poor performance. Good managers would find more incentive in simply doing good ideas, and allowing the markets to determine quality.

Someone from the other side

As a management consulting for one of the top three, let me point out two I would think to be pretty obvious things:
1) Many firms would not know how to setup experiments even if they wanted. It’s not just a mindset but also a talent thing. The longer I do this, the more I become convinced that talent is perhaps the scarcest resource we have
2) Many questions addressed to consultants are of the bet your company type. You simply cannot try and experiment, because the brick wall is approaching. Fast.

As for Joe, you are absolutely right, you cannot sell an idea to most companies with lets try – without clear management backing and according incentives, many people will do well whatever. There are few companies -mostly smaller, start up kind of ones and a few of the big research heavy juggernauts- who still do it, but even then, it’s not like “let’s try this” but more of “this seems to be in line with strategy, let’s see where we are with it in X, if we fail to achieve Z we will kill it”.

Someone from the other side

That should obviously read consultant. 🙂

Simon K

Engineering firms do experiments constantly. Otherwise nothing we did would ever work. I think the question is really “why don’t firms experiment on their customers and staff”? But this is a question analogous to “why don’t economists rely more on experiments”? The answers are the same – the obvious experiments are prohibitively expensive and damaging, the data obtained from “natural experiments” is unreliably and hard to interprety, and small scale feasible experiments often produce results that aren’t replicated on a larger scale (New Coke, anyone?). The effects are probably somewhat larger in management that in economics – firms care more about their reputations than economics professors (or even politicians), and data sets tend to be small and confidential to specific firms or even business units.

Grant

Given that most people outside of firms as well as other organizations prefer impressive consultants to experiments, shouldn’t a better title be “people won’t experiment”? Or do firms experiment less than other entities?

From where I’m sitting it looks like firms experiment more than other organizations. Of course they still may not do enough experiments.

I suspect that the expectation of security from following consultant’s advice is quite real. A modest degree of principal-agent problems allows a manager to take credit for success and blame consultants for failure.

John Maxwell IV

I think that sociology as well as economics plays an important role in understanding firm behavior. There’s a strong stereotype of corporate BS, and there must be reasons for this. The real question for someone like me is whether it’s possible to avoid this sociological denigration. From a Paul Graham essay:

At one point [Mitch Kapor and the head of HR at Lotus] worried Lotus was losing its startup edge and turning into a big company. So as an experiment she sent their recruiters the resumes of the first 40 employees, with identifying details changed. These were the people who had made Lotus into the star it was. Not one got an interview.

It seems likely that playing much closer attention to hiring decisions than usual might help a firm avoid sociological denigration, but this would probably also make it more difficult to grow.

What about governments? Why don’t they experiment? When they pass a trillion-dollar bailout or stimulus bill, do they even spend a billion dollars to study whether it worked or not?

We have some states that oppose a bill, and some that want it. That’s perfect! Instead of arguing over what every state in the union should do, we should argue over how to divide the states up into states that will implement the new law, and states that won’t, so that the experiment will be as well-controlled as possible.

This is a blog on why we believe and do what we do, why we pretend otherwise, how we might do better, and what our descendants might do, if they don't all die.