From Blackadder to Burgess and Maclean, this history of MI5 is a scholarly and hugely entertaining account, says Robert McCrum

Robert McCrum

The Observer, Sunday 11 October 2009

British intelligence officer and Soviet spy Kim Philby holds a press conference after being cleared of spying charges in 1955. Photograph: Getty Images

An authorised centenary history of MI5, the mysterious organisation whose existence was not even officially acknowledged until 1989, was bound to be a strange bestseller. But then, as Christopher Andrew amply demonstrates in this compendious volume, British countersubversion, founded in 1909 in response to Edwardian spy mania, stoked by a popular novelist and the Daily Mail, has always been a funny game.

The Defence of the Realm: The Authorized History of MI5

by Christopher Andrew

1088, Allen Lane

Andrew, a witty escort past the smoke and mirrors, divides his story into six parts, but really this is a game of two halves. Until the end of the Second World War, when Britain could still be classified as an imperial power, MI5 was chiefly engaged in taking on either “the Red Menace” or “the Boche” which ranged, in its peculiar taxonomy, from AA (“Absolutely Anglicised”), to BA (“Boche-Anglo”), to that nadir of subversiveness, BB (“Bad Boche”).

By 1917, the card index of suspicious persons held at the Security Service‘s central register numbered about a quarter of a million; the files were kept up to date by some 130 clerks, all women. One of the many surprises in this exceptionally long, but often enthralling, monograph is the discovery that Britain’s spooks have always been equal opportunity employers.

Since 1945, as the junior partner in the Anglo-American “special relationship”, a phrase Andrew celebrates at face value, MI5’s concerns have been driven by the defence of the west: the bleak skulduggery of the Cold War and, after 9/11, the need to combat al-Quaida. Today, counterterror, not counterespionage, has become the security service’s main priority and its female staff (it was formerly led by Stella Rimington, now a bestselling spy novelist) are said to enjoy a job satisfaction equal to the employees of Mills & Boon.

“Authorized” means what it says. Jonathan Evans, the current director general, recognises in his introduction that, in the age of public accountability, “we needed to commission an ‘open’ history for publication”. Almost in the same breath, he concedes that MI5 is “an organisation much of whose work must remain secret”. Andrew seems to have chafed at this restriction, referring darkly to one significant exclusion of material that is “hard to justify”. Much has been made of the 400,000 files he had access to at Thames House. The headline-grabbing revelations about Jack Jones and John Stonehouse (both communist agents) do not assuage the worry that the general reader will never know what’s missing.

Some flaws stand out. Possibly the most obvious weakness is the book’s treatment of Britain’s long, bloody and covert war in Northern Ireland, arguably the most significant peacetime countersubversion operation of recent times. To devote just five pages (in a volume approaching half-a-million words) to the Death on the Rock affair will strike some expert readers as a mite too “authorized”. Official history has it limits. The second half of Defence of the Realm, from 1945 to 2009, certainly breaks new ground. How much remains unploughed is not clear.

These are quibbles. Professor Andrew, who used to write for this newspaper and who also once taught this reviewer some French history, is an entertaining and authoratitive guide through the labyrinth of secret files, with an infectious fascination for the game of counterespionage, an insight shared by one of his subjects. “Running a team of double agents,” said one wartime spook, with the insouciance that the security service used to cultivate, “is very like running a club cricket side.” Professor Andrew is most at home in the breezy, Boche-hunting half of his narrative which reaches its climax in the brilliant Double Cross operations of the Second World War.

Turning German agents certainly had its lighter side. When one, code- named Summer, tried to flee on a stolen motorcycle to which he had strapped a canoe (for his planned cross-Channel escape), MI5 regional security went in hot pursuit, assisted by “some roadmen who stated that they had seen a man on a motorcycle, carrying a canoe, turn left down the Newmarket road”. Eventually, Summer was apprehended after he had fallen off his motorcycle in front of another navvy who had thoughtfully helped the errant Bad Boche “throw the canoe over a hedge”.

Farcical though these escapades might seem, the Double Cross activities that involved Agent Zigzag and Operation Mincemeat (the landing of a decoy military corpse in the south of France, to suggest a Mediterranean D-Day) exhibit the three immutable elements of MI5’s modus operandi.

First, there is the protection of our island security, the defence of the realm. Second, there is a kind of boy scout adventurism that involves invisible ink, cloaks and daggers, and the threat of poisoned ice cream. Finally, in a short step from code names, false identities and the willing suspension of disbelief, there is MI5’s exercise of make believe. The security service has attracted a remarkable cast of journalists, novelists and amateur fantasists. In its early days, it used to celebrate Christmas with a “Hush-hush revue”. Andrew is at pains to point out that part of MI5’s clubby mystique is its hospitable tolerance of “fun”, a characteristic that has also dragged it to that lowest point in its history, the failure of its operations against the Soviet Union.

In the secret state, disentangling fact from fiction is often impossible: in 1926, one agent penetrated a secret meeting of subversives in search of “a darkish fellow” named Blackadder. An important part of Andrew’s achievement is to narrate with clarity an incredibly complex story in which bizarre and improbable reality often outruns the most rococo fabrications of the spy novelist. Inevitably, at the secret heart of this book, lurk the infamous Cambridge spies, known to some in MI5 as “the Magnificent Five”.

Described by the KGB as “the ablest group of foreign agents in its history”, the spy ring of Philby, Burgess and Maclean (plus Blunt and Cairncross) became a humiliating corporate nightmare that permeated the middle years of MI5’s first century. Brilliant in its manipulation of German intelligence, especially before D-Day, the security service was comprehensively outwitted by the Russians. Andrew’s account of this fascinating episode, in which the Observer played a small part as Philby’s one-time employer, contains few revelations, but it does emphasise how, well into the 1970s, a generation of MI5’s senior management was traumatised by its failure to unmask “the Five”.

The backwash of paranoia swirled through every nook and cranny of government, including Number 10. Here, for the first time, is a comprehensive catalogue, and refutation, of Harold Wilson‘s ardent belief that he was the victim of an MI5 plot. In the 1970s, the Observer was among the first to report the allegations of “Pencourt” (Barrie Penrose and Roger Courtiour). Here, in all its delusionary madness, is the definitive account of Norman John Worthington’s (Wilson’s codename) suspicions

This faultline of paranoid fantasy runs on from Wilson to Peter Wright, the charlatan “witchfinder general” of Spycatcher notoriety. Professor Andrew does not like Peter Wright, and nor does MI5. Defence of the Realm is peppered with sharp comments against a maverick fantasist who, he argues, distracted the security service from its core purpose with his “vast conspiracy theories” and crazy speculations.

Here, the book develops its own agenda. More inter-office openness, a more reliable history and thus a better long-term perspective (from a historian) would, says Andrew, have improved MI5’s response to threats such as the Troubles. Incredibly, lacking any history or “perspective” about Ireland, MI5 and MI6 did not clarify which would take the lead in fighting the IRA until 1992.

After its struggles with the Provisionals, MI5 has been driven by history to develop into a counterterrorist organisation. Inevitably, in the ongoing war on Islamic terror, little can be reported, but the reader is left in no doubt that the defence of the realm is being vigorously conducted by the secret state with all the extraordinary powers at its command.

Much has changed. Recruitment is now conducted through national advertising. The identity of the DG (Jonathan Evans) is well known. By 2009, fact and fiction had morphed once again. The thoroughly modern MI5 is the model for the BBC’s Spooks. As devotees of that series will know, the spooks who matter are almost all women.