April 21, 2009

FT: An experimental approach to the right answers

An experimental approach to the right answers

Steven
Levitt has an unusual admission to make for someone who has just
finished teaching a class at the University of Chicago's Booth School
of Business.

"I don't know that much about business," says the economics professor whose 2005 book, Freakonomics , made him an unexpected household name.

"Four years ago, I especially didn't know much about business," Prof Levitt says, "but, when Freakonomics
came out, somebody decided to call it a business book - which it wasn't
meant to be - and, when you write a best-selling business book, you
automatically become a business expert. That makes businesses want to
talk to you - so, over the past four years, my colleagues and I have
spent a lot of time talking to businesses."

As an economist
suddenly thrust into the world of corporate America, Prof Levitt says
he was struck by the anomaly between economic textbook descriptions of
business behaviour and what actually goes on inside companies.

"What's
so strange to an economist who walks into a business is that economists
have a set of models that describe how businesses should optimally
respond. But that's not how businesses make decisions.

"That's
not to say the economists' models are necessarily right. The business
model can be sensible. It's usually pretty seat-of-the-pants, built
round a set of rules of thumb, but that makes sense because the world
is so complex and they have to make so many decisions that they can't
optimise every one. But there are some decisions that are too important
to make guesses on - and, in those cases, you either need to find data
to help you or to generate your own data through experiments."

That
view led Prof Levitt to team up with John List, a fellow economist at
the University of Chicago who has been conducting field experiments and
teaching experimental techniques for 15 years. Prof List says that,
after working with companies such as United Airlines and Chrysler, he
reached the same conclusion.

"The level of experimentation is
abysmal," says Prof List. "These firms do not take full advantage of
feedback opportunities they're presented with. After seeing example
after example, we sat down and said, 'We have to try to do something to
stop this.' One change we could make is to teach 75 to 100 of the best
MBA students in the world how to think about feedback opportunities and
how to think about designing their own field experiments to learn
something that can make their company better."

The two economists
decided to team up to develop a course for Booth students on "Using
Experiments in Firms" - the first time either had taught at the
business school.

They say data-gathering experiments at companies
fall into two broad cate-gories: accidental experiments - for example,
in which companies automatically generate data about how responsive
customers are to changes in prices - and those that are deliberately
designed and carried out, such as constructing an experimental
framework to find out what optimal pricing levels would be.

Although
they clearly favour the latter, Prof Levitt says they also want
students to recognise and analyse the accidental experiments their
companies are already undertaking. "The nice thing about accidental
experiments is that they're free.

"If you can figure out the
answer without having to design and conduct an experiment, that's
wonderful. But a lot of everyday activities that businesses undertake
could be transformed into experiments with almost no effort and almost
no cost. The way businesses operate more and more lends itself to being
able to run these real-world experiments. The lessons are enormous and
the costs are often trivial.

The intent is not necessarily to
churn out MBA graduates who will insist on conducting experiments when
they return to corporate life, says Prof Levitt, so much as to provoke
them to think about the issues.

"We teach the students that
there's a hierarchy of decision making," he says. "One thing you can do
is what's always been done. The second thing is you could just think
about whether what you're doing is the right thing. The third step in
the hierarchy is you could take the data you have and try to analyse
it. But the problem is that it's often not the right data. The fourth
rung would be an accidental experiment. The fifth would be to go out
and generate the idealised data you would like to have."

Realistically,
companies may not have the resources or the know-how to design
experiments and analyse data, so when they look for information, they
often turn to outside experts. Prof Levitt says he impressed on the
students that they should be sceptical of such analysis.

"Although
it seems incredibly simple, even analysing the data trips people up and
the way you design the experiment affects how much data you get out of
it and how generalisable those data will be to other settings," he says.

Prof
Levitt gives the example of a company that was told by consultants that
sales surged when it advertised on television - without realising that
its advertisements appeared just before big holiday sales days. "The
consultants had missed an obvious point," he says. "It wasn't that
advertising caused their sales to go up but that knowing their sales
were going to go up caused them to advertise a few weeks earlier."

The message seems to have got through.

"We
learnt not to take data at its face value," says Hannah Levine, one of
the Booth students on the course. "You really have to understand what
tests were run to test the hypothesis and work out whether the analysis
was correct."

Prof Levitt suggests that the lesson is just one of
a number of fresh perspectives that he and Prof List have brought to
the Booth MBA. "Even if our students don't become experts at
data-crunching, at least they've thought about how you can be misled
with data, how to think about correlation and causality - issues that
have tended not to be central to an MBA's education."

Meghan
Sheehan, who will join the Global Leaders Programme at Barclays Bank in
London after she graduates from Booth this year, says she can imagine
using the techniques she learnt to look at issues such as how the bank
can discover the best way for its customers to keep in touch.

"You
could design experiments to test the relative virtues of telephone,
internet and branch visits," says Ms Sheehan. "How could you
incentivise people to bank online rather than in a branch?

"Do you use a negative incentive to make in-person banking more difficult or do you offer an added benefit?"

These
are exactly the sorts of questions Profs List and Levitt are trying to
provoke. "We're on a proselytising mission of bringing a different way
of thinking," says Prof Levitt.

"We're trying to bring about a
revolution in business, so this is the first shot over the bows. This
is an important battle to change the way businesses think."

Changing business thinking means placing datagathering at the heart of business practice.

"With the right data, you can improve your decisions," says Jasper Platz, another Booth student.