With endless supplies of Pol Roger champagne, the Spectator's annual parliamentary awards ceremony at Claridge's tends to be a lively event for Britain's political elite.

Last November's lunch proved no exception. As immaculately dressed waiters served up roast lamb in the hotel's chandeliered ballroom, one of the prime movers in London politics gave vent to months of frustration. "Boris, you've got to pull your finger out," the Evening Standard editor, Veronica Wadley, told the Tory candidate for London mayor.

An Eton scholar who does not take kindly to being rebuked, Johnson mumbled incoherently as other guests took the opportunity to air their own concerns about his lacklustre campaign. But the blunt assessment hit home. By the turn of the year, his campaign had sharpened up and now Johnson could be close to the biggest political breakthrough by the Conservatives since John Major's surprise general election victory in 1992.

"Boris looked very sheepish as Veronica told him he had to pull his finger out," one of the guests said. "A lightbulb came on - by January he was really energised and now he may be on the verge of an extraordinary upset."

David Cameron, who faced criticism last year for throwing his weight behind such a maverick, now believes he may soon be vindicated. One senior Tory explained Cameron's thinking. "Ken Livingstone is one of the most skilled campaigners in British politics. And yet Boris is now slugging it out with Ken and setting the policy agenda of the campaign by focusing everyone on crime and transport. Even if Ken pulls it off - and Boris is within striking distance - that would still be a pretty stunning result for us."

A strong Johnson showing would provide the first electoral evidence that the modernised Tory party is reaching far beyond its usual comfort zones. "London is totemic and will grab the headlines on May 1," said one member of the shadow frontbench.

"This is the most multiracial and multicultural city in the country. If we can do well there that will really capture people's imagination."

If Johnson wins, the debt he will owe to Associated Newspapers, and Wadley in particular, will be huge. For the first time, the capital will have a mayor in tune with the Standard's concerns on crime, the congestion charge and the costs of the Olympics.

There is another factor: in two years' time, Associated's multimillion-pound contract to supply tube stations with the free Metro newspaper will expire. The renewal of the contract is a decision for Transport for London, whose chairman the mayor appoints.

A nod of thanks will also be due to Rupert Murdoch, whose Sun came out for Johnson this week, hailing him "a new and fresh champion for London".

But he was by no means an automatic choice for the Tories. And he needed to be persuaded to stand. Wadley played an important role in championing him last summer, and it is her newspaper that has provided him with unswerving support ever since.

"Veronica told anyone who would listen in senior positions in the party that there was only one guy who could win this: Boris," one senior Tory said. "That had a profound effect on David Cameron and George Osborne."

Johnson had flirted with the idea of standing for London mayor for some time, but it took a light-hearted intervention from a former boss - and a pillar of the Tory establishment - to make up his mind. A friend sent an email last year to Charles Moore, the former editor of Daily Telegraph, where Johnson first made his name as a journalist. It read simply: "Boris for mayor?"

The mischievous Moore fired back a reply: "Boris for mayor of Henley" - the affluent Oxfordshire constituency he represents in parliament. "Boris was so outraged by what Charles wrote that he was spurred into action," said a friend.

The sight of a rejuvenated Johnson at the end of the year came as a relief to the Tory leadership. Last autumn, they were seriously worried at the faltering start to his campaign after he was overwhelmingly selected as the Tory candidate on September 27.

Osborne, the shadow chancellor, passed on his concerns to Johnson at about the same time as the Wadley ticking off at the Spectator lunch. Johnson listened carefully to the man who will coordinate the Tories' next general election campaign; he has learnt to respect Osborne, who was made the Spectator's politician of the year at the Claridge's lunch.

The Osborne intervention - and the pressing need to get their candidate back in the race - led the Tory high command to seek out someone who could both discipline Johnson and identify and exploit the issues that concerned Londoners.

Their search led them to Lynton Crosby, the Australian strategist who delivered a string of election victories for the former Australian prime minister John Howard.

Despite Crosby's failure to secure a win for the former Conservative leader Michael Howard in the 2005 general election, Osborne, Cameron and the chair of the Tory party, Caroline Spelman, decided he was the one to break in their candidate.

Since Crosby took charge in January, Johnson has learned to talk seriously about issues such as crime, public transport and housing - and toned down the buffoonery. Crosby's decision to focus on outer London boroughs such as Enfield and Bexley, normally neglected by mayoral candidates, and to pinpoint wards where the Tories should concentrate their efforts, have also won him praise.

One senior figure said: "Lynton has worked out the outer London strategy. He has got armies of people out to tube stations. A lot of themes that he used for our 2005 campaign are now being used. But this time we have the right messenger in Boris and David Cameron has made it possible for the party to talk about these things."

The Crosby campaign has appointed an outside press team - iNHouse PR - run by the former Tory press officer Katie Perrior. Ed Staite, a popular Conservative press officer who is expected to move to City Hall with Johnson if he wins, is also on hand. In a sign of the Australian-style approach to the campaign, Perrior is on a commission - the better Johnson performs the more her company will be paid.

Whenever Johnson takes to the streets, be it Soho for the Chinese new year or Edmonton for his campaign launch with David Cameron, his PR minders are never more than an arm's length away, appearing to act as much as bodyguards as gaffe-guards. They often round on journalists who fail to portray Johnson in a flattering light - even those who work for "friendly" publications such as the Evening Standard.

Crosby himself also favours the hands-on approach. At both the unveiling of Johnson's crime manifesto at Millbank in February and the Edmonton launch, he looked on mutely from the sidelines as if trying to transmit telepathic messages to his protege. And while he will exchange a few words with journalists, his gaze never seems to leave his potentially unruly charge.

Some believe that Crosby has not so much banished Johnson's outer clown as coaxed out the old Etonian's inner self.

Lloyd Evans, the Spectator's theatre critic, who has known Johnson since they were at Oxford, said his friend - now "a bit plumper" than at university - was finally coming across as "the fully rounded Boris".

Johnson still enjoys the support of a number of Oxonians - including a few members of the Bullingdon club, the dandyish dining society to which both he and Cameron belonged. A handful of former Bullingdon members attended a fundraising dinner hosted by Johnson at the beginning of this month.

But despite the overhaul, support and some intensive cramming - he has asked more than 100 parliamentary questions about London in the past few months - Johnson is often still visibly struggling to master his brief.

At a KPMG business hustings at the end of March, he dithered over whether or not to shut down the mayor of London's overseas offices, saying he needed to consult the business community. When it was pointed out to him that he was addressing the business community, and that most of its members favoured the "embassies" and their trade benefits, he dithered a little more. Last week he went back on his original pledge to close them. Similar indecision over the cost of his plans to replace the bendy buses he loathes with a new generation of eco-friendly Routemasters has been one of the most salient and embarrassing features of his campaign.

After initially admitting that he did not know how much the buses would cost, he alighted on a figure of £8m.

The low figure was immediately questioned by independent transport experts, who suggested he was out by around £100m. Nonetheless, Johnson stuck to the £8m figure until last week, when he was caught on a mobile phone camera telling a Labour supporter that he believed the scheme would cost around £100m.

To muddy the waters still further, he appeared on a BBC debate a day later claiming that the new buses would cost no more than £8m. He now estimates the cost to be around the £100m mark.

His ability to weather this series of policy fumbles is proof of the goodwill his personality seems to evoke from the public and certain sections of the press: had Livingstone shown such a hazy knowledge of his key policies, he would have been torn to shreds - particularly by the Evening Standard.

The success of the past three months has left the Tory high command pondering what Mayor Johnson could mean for the party nationally. Would he rally young non-committed voters with his Woosterish japes, or would he allow Labour to paint the Cameron Conservatives as unreliable and unsuited to office?

Opinions are divided among Tories, though most believe the Johnson gamble was worth taking. One senior figure spelled out the thinking of those who have yet to be convinced.

"One of the reasons why Boris was encouraged to go for it is that he is a semi-detached member of the Conservative party. He is of the Conservative party but not with the Conservative party. His relationship with the Conservative party is a bit like London's relationship with the rest of the country. That is why he is such a good call.

"But I do not think we have got our heads round what a Boris win will mean. Boris has this manner that makes it like entertainment rather than serious campaigning. Conservatives will be as unprepared as London if Boris wins.

"There will be a feeling, this could be interesting, it seemed a good idea, but are we ready for this? Boris will be his own man. How will he relate to the Labour government? ... This is a double-edged sword. A victory would say the Conservatives are coming back. But there is also the element that you can't control Boris. If he screws it up that will rub off on us."

Johnson's screw-ups - or gaffes as they are more charitably known - are legendary. In the course of his career, he has referred to black children as "piccaninnies" in an attempt to mock Tony Blair's globe-trotting, and the people of Papua New Guinea as cannibals. He even sympathised with parents who did not like Jamie Oliver's campaign to make school meals healthier.

He eventually apologised to the black community and acknowledged that some of the things he had written had been offensive, but many black people remain wary of Johnson. Among them is Doreen Lawrence, the mother of the murdered teenager Stephen Lawrence, who told the Guardian last year that Johnson was "not an appropriate person to run a multicultural city like London". He was initially booed at a debate earlier this month organised by London Citizens, an umbrella group for scores of religious and ethnic minority groups. Although he went on to charm most of the crowd, when a black woman approached him after the event to berate him for some of the things he had written, he was swiftly surrounded by aides and whisked away.

Johnson thinks the charges of insensitivity about race are unfair. He likes to point out that his great-grandfather was Turkish and his wife, Marina, is half Indian.

While old articles have come back to haunt Johnson, his way with words, when properly focused, has served him well, propelling him on to the leader writers' desk of the Telegraph at the age of 23, and into the editor's chair of the Spectator 12 years later. He has also written a handful of books, including a novel, and presented a TV series about one of his passions, the Roman empire.

Another senior Tory was enthusiastic about Johnson's chances, pointing to his intellectual rigour and determination.

"We are all surprised to find ourselves in the position where Boris is the frontrunner. Ken Livingstone has held the London franchise for a generation. We all thought it was a long shot to win it. I always thought that to win it we needed a candidate who was a big personality. I have always thought that Boris is a serious figure. He is very hungry and he is driven."

The prospect of a Johnson victory has led senior figures in the party to make military-style preparations for a new high-flying City Hall team. Nick Boles, the founding director of the modernising Policy Exchange thinktank, is vetting candidates for up to 12 posts such as chief of staff to a Mayor Johnson. Boles, whose main job is to prepare members of the shadow cabinet for life as government ministers, would have been the Tory candidate for mayor had he not fallen ill a few years ago. He is now fully recovered.

One frontbencher said: "Nick is working out what responsibilities each person would have at City Hall. He is working out whether you need to re-engineer the posts held under Livingstone by [former race adviser] Lee Jasper and [chief of staff] Simon Fletcher. The mayor has a lot of personal appointments." Johnson used a mayoral debate last week to announce that Bob Diamond, president of Barclays Capital, would advise him on encouraging City firms to invest in community projects. Diamond is the highest paid FTSE 100 boss, earning £36m last year. On Monday, Johnson revealed that other advisers to the Mayor's Fund for London would include the former Labour donor Sir Trevor Chinn and his mayoral campaign fundraiser, Lord Marland. Steve Norris, the Tories' previous candidate, expressed an interest in running the London Development Agency but added that nothing had been agreed.

The leadership and Johnson are determined to keep a lid on the people they have lined up for City Hall because they have other jobs which they may need to hold on to if the Tories lose. But if Livingstone overtakes Johnson in the closing days of the campaign, the Tories may release more names to show he is capable of attracting talent to his team.

Johnson's critics are keen to point out that the Conservative challenger has never represented anywhere bigger than the safe Tory seat of Henley, which has little in common with London but the Thames. His constituents, though, seem pleased with his performance and he enjoys a majority of almost 13,000.

As he stood on the steps of Henley town hall three weeks ago, waiting to begin his constituency surgery, a stream of locals stopped for a chat or honked their horns as they drove past. He shook hands with one enthusiastic constituent who wanted to wish him luck in London.

"Thank you very much," he said. "But it's going to be very close."

Looking out over his parliamentary fiefdom, he seemed relaxed and more like the Johnson of old. He shrugged his shoulders over the news of Livingstone's "secret" children, insisting it would be irrelevant to Londoners, but complained grumpily that the mayor seemed set on stealing his policies.

Asked how he felt about the BNP advising its supporters to give him their second-preference votes, he reached for Virgil: "Non tali auxilio nec defensoribus istis tempus eget" (Not such aid nor such defenders does the time require).

And, before mooching off to Oxfam to browse the secondhand books, he had a confession to make about his highly motivated PR people: "They scare me, too."

Standard views

Some stories from the Evening Standard this year. The newspaper is edited by Veronica Wadley.