Saturday, 2 March 2013

How fast do institutional changes take effect?

Today I saw a presentation of a field
experiment evaluating different ways to deliver aid. One was the
standard method which had been used until that point. The other two
“treatments” were new. The evaluation was over the course of a
year. (There were lots of pictures of smiling villagers... lab guys
don't have those.)

How long do we have to wait before new
institutions settle down and we can evaluate them “in equilibrium”
– meaning, not any rigorous game-theoretic concept, but just that
people have somehow got used to them, and that all the changes have
worked through the system? In this case I felt that a year was too
little. Aid recipients are unlikely to be naïve about the fact that
new institutions are being tried and evaluated, and they may
therefore behave in a special way in the first season of a change.

I worry in general that social
scientific evaluation is too short-termist, and that the tools of
statistical analysis can encourage this. For major institutional
changes, there is a good case that the smallest possible
“independent” unit of observation is a generation. Until people
have grown up under a new system, we are not sure that its full
effects have been worked out.

In this context, advanced statistical
analysis can actually be a step backward. For example, consider this paper which estimates the effect of democracy on GDP – a
traditional hobby for political scientists. Now as everyone in the
field knows, just looking at democracies versus dictatorships and
comparing averages will not be informative, because these countries
differ in many many other ways. So instead the paper looks at the few
years before and after a change from dictatorship to democracy, and
estimates the switchover effect from that. But this is crazy, because
such massive changes to social institutions are not remotely likely
to have all their effects within a few years. After all, many
political decisions have ramifications that span decades – think of
the choice to create the NHS, or Lloyd George's introduction of old
age pensions, or Nixon's visit to China. Evaluating political
institutions after seven years is like evaluating a new fitness
regime after a week.

I can think of several cases where my
previous beliefs were probably based on too short a run of evidence.
For example, I assumed that privatization of e.g. water utilities was
a good thing, because the privatized utilities performed better than
public counterparts elsewhere. But a lot is going to depend on the
first generation of entrepreneurs who take over, and these may not be
the same as the second or third generation of entrepreneurs who
inherit the system. In the 1950s, nationalization must have seemed as
obvious as privatization did in the 1990s. I am not saying that
privatization was a mistake – I still support it – but I am less
confident of the evidence base.

My colleagues seem to be making a
similar mistake about the effect of the Research Assessment Exercise
(now the Research Excellence Framework) for UK academia. Everyone I
know who was around in the 80s, when this came in, says that it
swiftly forced a lot of unproductive, “dead wood” academics to
either shape up or leave the system. So they are basically positive
about it. (Well, Essex political scientists would be, wouldn't they?)

The question is whether it is still
having the same effect now. When a new institution is imposed, there
are two kinds of adjustment: people adjust to the institution; and
the institution is adjusted to the people. After all, nobody wants to
live under permanent revolution, so initially harsh conditions are
gradually softened, informal routines grow up that may subvert the
official rules, et cetera.

In this context it is pretty alarming
to consider the Conservatives' and then New Labour's regime of
targets for the NHS (known by some as “targets and terror”).
Again, I have heard people in the industry talk about the salutary
initial effects of having managers asking “why isn't this bed being
used”? But now look where targets and terror have got us.

If this argument is right, we will
often be unable to evaluate institutional changes rigorously until
long after the fact – even if we are doing randomized controlled
trials, which is often impossible. So, how can we decide what changes
to make in the here and now? Two things might help.

First, a historical perspective won't
tell us what will happen, but will at least give us a sense of what
can happen. Without history we are doomed to parochialism. As
Churchill said, "Study history, study history. In history lies all the secrets of statecraft."

Second, a sense of principle might
often be a good guide. Actually, another Churchill quote is relevant: "In life the only wise course is to follow the course of duty and not of interest. Every man knows what his duty is. But it is not given to many to know their true interest." To apply this to academia: we may not know, now or ever, the
true efficiency effects of such-and-such a government evaluation
framework, or of the practice in an increasing number of European
universities of – no joke – paying bonuses for top
journal publications. But every researcher should feel that the
search for truth is sacred, that it requires rigorous and demanding
standards of honesty in the muddy waters of empirical analysis, and,
therefore, that attempts to import monetary incentives, or impose
pressure to publish, should be met with great suspicion.