Those who have long detected a frisson of rivalry between Boris Johnson and his fellow Old Etonian David Cameron should turn to page 295 of Sonia Purnell’s biography of the London mayor. There we learn of Johnson’s weekly struggle to find subjects for his lucrative newspaper column. When stuck, he eggs on his staffers to suggest themes guaranteed to destroy his career. Johnson’s favourite? The one on which he riffs with the greatest gusto? “Why David Cameron is a complete c**t.”

Purnell’s book bristles with such stories, all illustrating Johnson’s high-wire approach to politics. He knows no caution. In his early days as a Tory MP, when still editor of The Spectator, the magazine’s cover showed Michael Portillo peeing on the head of his party leader, Iain Duncan Smith. When Cameron talked up the idea of “broken Britain” a few years later, Johnson cheerfully declared his boss was talking “piffle”.

Lesser offences have buried the careers of many. That Johnson has not only survived but prospered shows his unusual position in the political firmament. True, he has yet to hold an office of any great power. Strip away the showbiz that surrounds the mayoralty of London (a post he won in 2008) and, in effect, you have a glorified local official responsible for transport.

But the mayor’s showbiz can tickle parts of the electorate that policy wonks can’t even name. As the Tories gather for their conference, many activists still see him as a leader-in-waiting. Johnson may have rated his own chances of reaching Number 10 as “about as good as the chances of finding Elvis on Mars, or of my being reincarnated as an olive”. But in an age of focus-grouped, Photoshopped politicians, he could conceivably go to the very top – unless he trips himself up first.

And trip he will. He generally does. For along with the Edwardian manner, the Bertie Wooster vocabulary and the tousled charm comes the temperament of the gentleman amateur. Johnson, as Purnell shows, has spent his life winging it. His talented languor frustrated his masters at Eton. In Brussels in the 1990s, he alienated fellow hacks by his cartoonish exaggerations. When he left, a colleague parodied Hilaire Belloc’s Matilda for his leaving party: “Boris told such dreadful lies / It made one gasp and stretch one’s eyes ... ”

Johnson’s survival depends on a genius for being let off. He is indulged partly because of his talent but also, Purnell speculates, because of a curious vulnerability others sense in him. He is forgiven as a child is forgiven. One of his former editors compares him to a “hyper-intelligent two year old” who has yet to understand the world does not revolve around him.

Undiluted solipsism can be a strength in a politician. And Johnson’s comically self-promoting projection has secured him a remarkable connection with the British public. In an age of deep cynicism towards politicians, he is genuinely popular. The American journalist who once described him as the inheritor of the natural charm and good humour of Ronald Reagan had a point.

For all the japes about Elvis and olives, he is in deadly earnest about getting to the top. What is interesting about his rivalry with Cameron is the understated, crab-wise manner in which Johnson attacks – it has been called “drive-by politics”. So he naughtily singles out the crime of “wisteria clearing” (the one claim for which Cameron got into trouble) when asked to comment on the MPs’ expenses scandal. Purnell roots this in Johnson family lore. His great-grandfather, Turkish politician Ali Kemal, was lynched after backing the wrong side in the civil war that followed the dissolution of the Ottoman Empire. Every Johnson, she says, is told this story as a warning of the perils of direct confrontation.

But if he clearly hungers for power, it is less clear what he wants to do with it once attained. While expert at milking the London job for photo opportunities, his achievements are thin. Thanks to his celebrity and colourful past, Johnson has largely avoided real scrutiny at City Hall. “People like [broadcasters Jeremy] Paxman or Andrew Neil just ask him about the Bullingdon club or what he thinks of Cameron,” points out one insider. “He knows they won’t press him on the real stuff such as spending cuts and U-turns.” His deeper political beliefs remain a blank page.

This is a thorough biography and Johnson is a compelling subject. But the code is never really cracked and occasionally Purnell resorts to unsatisfying, cuttings-fuelled speculation. The book crackles into life only when its subject, a natural comic, is quoted. To that extent it illustrates his position in British public life. Johnson – blond, absurd, risk-taking, solipsistic – is in blazing colour while his rivals are in black and white.