29 April 2017

Forecasting

“The study of history, while it does not endow with prophecy, may indicate lines of probability.” John Steinbeck

“I always avoid prophesying beforehand, because it is much better policy to prophesy after the event has already taken place.”
Winston Churchill

In an age when it would seem that the people will be asked to speak out on the future of the nation at least once a year, apparently in the interests of stability, the natural tendency is to seek out the opinion of experts who can tell us what will happen. This is despite the clear evidence over some time now that most of these experts don’t know. For me the result of Brexit was clear months before it was announced. I wrote to the Prime Minister in March of last year advising him to prepare to write his memoirs. I think I should have asked for commission on this advice as he is reported to have signed with William Collins, the publishers for £800,000, though very much less than Margaret Thatcher or Tony Blair.

You see as a marketing man I am not just interested in whether people like my product. I want to know how much they like it. We called this ‘the hedonic scale’ and new products would not be launched unless they achieved a certain minimum based on decades of experience. And I was not just interested in whether they said they would buy it, but how likely this really was given their other purchasing habits. We called this ‘propensity to purchase. ‘ And with these two scores we could test the product in the market, measure repeat purchase and so derive forecasts of how much we would sell on national launch.

When someone is asked by a pollster to answer how they would vote in a referendum on Britain remaining in or leaving the European Union you might get a score of 52% remain, 48% leave. So most people were surprised when the actual result was the other way round. Many were shocked because they had cut themselves off from the diversity of public opinion by staying in their social media bubble. The equivalent of ‘hedonic ‘scale and ‘propensity to purchase’ needed to be applied to the raw data. And it was clear to me that many more people really cared about leaving than remaining. Many of those who voted to remain did so out of fear of the unknown rather than a passionate desire to stay in a creaking institution incapable of reforming itself.

There is no shortage of forecasts. We are surrounded by them. Overwhelmed by them. The news sometimes seems to consist of little else. We are told that the IMF has revised its forecast for the UK’s economic growth, not that it is nearly always wrong. Why does anyone listen to them or report their worthless forecasts?

Let’s take some extreme examples:

In September 2001 Islamic terrorists flew hijacked planes into the World Trade Centre. US intelligence said this was the work of al-Qaeda (almost certainly true), led by Osama bin Laden (true), who was being harboured by the Taliban in Afghanistan (true.) So the US invaded Afghanistan to oust the Taliban. but then for some reason the administration of George W. Bush turned its attention to Saddam Hussein’s Iraq; suggesting that Iraq had ties to al-Qaeda (almost certainly untrue), that Iraq was therefore behind 9/11, (almost certainly untrue), that Iraq was a menace to other countries in the Middle East (true viz, Kuwait), that Iraq had not destroyed its WMDs as required by the United Nations, (almost certainly untrue), that it was building up its stockpiles (almost certainly untrue) and was becoming more dangerous by the day.

Then on October 2002 it made the following public statement: “We judge that Iraq has continued its weapons of mass destruction (WMD) programs in defiance of UN resolutions and restrictions. Baghdad has chemical and biological weapons; if left unchecked, it probably will have a nuclear weapon during this decade.” (My emphasis)

If you parse this then a judgement might imply a judgement of probability and the final assertion is clearly a statement of probability, though in neither case is a percentage applied to the probability. However, the middle statement, which I have emphasised in bold, is an assertion of fact, i.e. of 100% probability. Following this the administration proceeded to make the same assertions to the UN through the highly credible representation of Secretary of State Colin Powell and built a case for a disastrous war. No WMD were found. Hundreds of thousands died. And the ‘country’ of Iraq remains in chaos.

This is not a blog about the politics of this but about the reliance on unreliable forecasters in making far reaching decisions. The US intelligence community has a budget exceeding $50 billion and employs 100,000 people. But it did not say we think there is a good chance that Saddam Hussein is still developing WMD. It stated he had chemical and biological weapons.

Around this time a certain Donald Rumsfeld was Defense Secretary. On April 11, 2001, he sent a memo to President George W. Bush and Vice President Dick Cheney. “I ran across this piece on the difficulty of predicting the future, “ Rumsfeld wrote. “ I thought you might find it interesting.” The piece examines the strategic situation at the start of each decade between 1900 and 2000 and shows that, in every case, the reality was dramatically different from ten years before.

“Thoughts for the 2001 Quadrennial Defense Review

· If you had been a security policy-maker in the world’s greatest power in 1900, you would have been a Brit looking warily at your age-old enemy, France.
· By 1910, you would be allied with France and your enemy would be Germany.
· By 1920, World War I would have been fought and won, and you’d be engaged in a naval arms race with your erstwhile allies, the U.S. and Japan.
· By 1930, naval arms limitation treaties were in effect, the Great Depression was underway, and the defense planning standard said “no war for ten years.”
· Nine Years later World War II had begun.
· By 1950, Britain no longer was the world’s greatest power, the Atomic Age had dawned, and a “police action” was underway in Korea.
· Ten years later the political focus was in the “missile gap”, the strategic paradigm was shifting from massive retaliation to flexible response, and few people had heard of Vietnam.
· By 1970, the peak of our involvement in Vietnam had come and gone, we were beginning détente with the Soviets, and we were anointing the Shah as our protégé in the Gulf region.
· By 1980, the Soviets were in Afghanistan, Iran was in the throes of revolution, there was talk of our “hollow forces” and a window of vulnerability”, and the U.S. was the greatest creditor nation the world had seen.
· By 1990, the Soviet Union was within a year of dissolution, American forces in the Desert were on the verge of showing they were anything but hollow, the U.S. had become the greatest debtor nation the world had ever known, and almost no one had heard of the internet.
· Ten years later, Warsaw was the capital of a NATO nation, asymmetric threats transcended geography, and the parallel revolutions of information, biotechnology, robotics, nanotechnology, and high density sources foreshadowed changes almost beyond forecasting.
· All of which is to say that I’m not sure what 2010 will look like, but I’m sure that it will be very little like what we expect, so we should plan accordingly.

January 9, 2001 Lin Wells

History does not record how Messrs. Bush and Cheney reacted to this incisive memo, but it does record how they did not make the world or the US a safer place. But how do you prepare for a future where it is so difficult to forecast with accuracy. Well, in politics Winston Churchill had it right as usual where he defined political ability as: “It is the ability to foretell what is going to happen tomorrow, next week, next month and next year. And to have the ability afterward to explain why it didn’t happen.”

I found Lin Wells’ memo in an excellent book by Philip Tetlock and Dan Gardner called Superforecasting.[i] Philip Tetlock is the researcher responsible for a twenty year exercise where he gathered a large group of experts – academics, pundits and the like – to make thousands of predictions about the economy, stocks, elections, wars, and other issues of the day. Time passed, and when he checked the accuracy of his predictions, he found that the average expert did about as well as random guessing. This was widely reported as: the average expert was roughly as accurate as a dart-throwing chimpanzee.

But he has gone on to run an even bigger study that does not recruit so-called experts, but rather ordinary people who may be out of work or retired and have the time to study situations and make predictions about them. They correspond with each other, they constantly make minor adjustments to their forecasts and they produce significantly greater accuracy, hence his nickname for them, the Superforecasters. The book is quite technical in places but is still rewarding for the myriad examples of where experts have been proved wrong and these part-timers right.

But thinking about it as wrong or right is not always helpful. Take one example of where the forecaster is often pilloried for being wrong: the weather. When a weather forecaster delivers his forecast he is telling you correctly what the likely weather is going to be. It is not possible to say he was wrong when he is telling you the probability of something. To find out if he was wrong you would have to run some gigantic Groundhogs Day meteorological experiment and run the same conditions for numerous days until you got an actual measure, clearly an impossibility.

So the sensible person prepares for the likely weather and its range of possibilities. It may rain so I take an umbrella. It may get cool later so I’ll take a coat.

And that’s how you deal with future possibilities, whether its politics or economics, whether its business or other issues of major importance. Or whether it’s about the risk of rain.