From the dizzying top of the Shard, the Palace of Westminster is a distant blur. But few of the peers and MPs attending a reception hosted by the ambassador of Qatar on Wednesday night found time to give more than a casual glance to the stunning views.

Crowded on to the viewing platform of Britain’s tallest building, 1,016 feet above the capital, they huddled in corners and snatched whispered conversations over the canapés. Among those present were Alan Duncan, the dapper and well-connected minister for international development, along with Bill Wiggin, an Eton contemporary of David Cameron, Andrew Rosindell, the Right-wing campaigner from Essex, and Andrew Robathan, the defence minister and former SAS officer who was among the first to back Mr Cameron’s leadership bid in 2005.

In keeping with the traditions of the Gulf emirate, no alcohol was served, but the politicians needed no encouragement to loosen their tongues and trade speculation about the only subject galvanising Tory politicians this week.

Anyone eavesdropping would have heard, carried on the wind, the name Adam Afriyie repeated again and again, passing from one group to another. To one of those present, the scene recalled the conspiratorial huddles in the Winter Gardens that marked the 2003 Tory conference in Blackpool that precipitated the fall of Iain Duncan Smith.

Quite how a little-known MP who has left few footprints since he entered the Commons in 2005 has kicked off such a frenzy of speculation about Mr Cameron’s future is a study in the current febrile state of the Conservative Party at Westminster.

The public may see a Conservative Prime Minister going about his official business – this week on a trip to Africa – untroubled by any obvious party concerns. But many of his MPs see a leader who is not a winner, and whose actions – and inactions – are leading the country to disaster and the Tory party to extinction. Away from the public eye, some are actively working to bring him down, while others watch and wait, quietly hoping for his demise.

It is a measure of their discontent that even those closest to Mr Afriyie’s leadership bid admit privately that he stands no chance of actually becoming leader. Rather, they hope that his campaign will serve to bring down Mr Cameron and clear the way for someone else. It is this “John the Baptist” scenario that has everyone talking at Westminster: is Mr Afriyie merely preparing the ground for a more credible candidate?

There has always been talk of who might succeed Mr Cameron in the event of an ''under the bus incident’’. William Hague, Liam Fox and David Davis are talked about, with differing degrees of enthusiasm. The Foreign Secretary has given every indication that he has no intention of seeking the top job. Dr Fox is ubiquitous as a champion of the loyal Right, while Mr Davis has become one of the Prime Minister’s most persistent critics.

Among others in the Cabinet who have let it be known that they would be available if the call comes are Theresa May, Philip Hammond, Owen Paterson, Michael Gove, Chris Grayling, and Jeremy Hunt. Further up the Tory food chain, the prospects of Boris Johnson and George Osborne are much discussed.

Younger Tories, especially those first elected in 2010, suggest that if and when a vacancy arises, it must be filled not by a veteran but a fresh face. Jesse Norman, the thoughtful MP for Hereford, is tipped by several backbench colleagues.

For many, this amounts to little more than idle speculation over dinner. But that alone tells us that Mr Cameron’s position is far from secure.

Mr Afriyie, a self-made millionaire who founded his own IT business, has been carefully cataloguing signs of Tory discontent. Friends say he has a database of MPs prepared to defy Mr Cameron and possibly put their names to a letter demanding a vote of confidence.

His work has been underway since last summer. In particular, he has focused on MPs sitting for marginal seats who may be growing increasingly anxious about their chances of surviving the next election. “It’s the 'no-change, no-chance’ group,” one says. “Those who believe that they are doomed with Dave.”

Mr Afriyie’s allies insist that far from being a pawn in someone else’s leadership gambit, he has been working for months to build up an organisation. He has told friends that he wants Mr Cameron ousted before 2015, is preparing for that eventuality, and has something to offer his party.

Since last summer he has quietly worked his way around the party, seeking out senior figures whom he hopes might view his candidacy with sympathy. Supporters say he has spoken to more than 100 colleagues, but only a handful have so far signed up.

One is Mark Field, the Tory MP for the Cities of London and Westminster, who this week insisted that far from threatening Mr Cameron, Mr Afriyie was merely preparing for the eventuality that there might be a leadership vacancy after the next election. Mr Wiggin, breaking his silence yesterday to laud Mr Afriyie’s leadership qualities, sings a similar song.

Yet if few are willing to back him as leader, many are silently willing him on, hoping he will be the catalyst for change. “I’m not backing him yet, but I’m happy he’s doing this,” says one government member. “We can’t go on like this, and if this is what it takes to get the message through to No 10, so be it.”

That MP is one of many who privately admit they have lost confidence in Mr Cameron. They are frustrated by a range of things, from the party’s weak standing in the polls to Mr Cameron’s newfound desire to involve Britain in military adventures in Africa.

Underlying all this is widespread unhappiness with Mr Osborne’s failure to deliver economic growth or substantial spending cuts. MPs claim that with public spending and debt levels still rising, the Coalition has left itself with the worst of all possible worlds: no growth and ineffective austerity.

There are also persistent complaints about the way Downing Street operates. A succession of misfired briefings on child tax allowances and military spending has been laid at Mr Cameron’s door. “Shambles is what we stand for,” one MP says.

MPs who have been involved in the conversation about removing Mr Cameron point out that securing the 46 signatures needed to force a vote of confidence is not enough. Although the PM requires a simple majority to survive, dissidents believe the opposition of 100 or so MPs – a third of the party – would be a fatal blow to Mr Cameron.

For now, the would-be assassins believe they are well short of that mark, so a confidence vote is far away. But others believe that discontent in the party is so deep and wide that an endgame could come about quickly and dramatically, even accidentally.

The May local elections, the next GDP figures, a scandal arising from Mr Cameron’s old aide Andy Coulson, a military disaster in Africa; all are adduced as potential triggers. Then there are the unexpected twists, the unforeseen events that so often determine the course of political history.

“There’s more than enough dry tinder under the leadership,” says one senior Tory. “It would just take one spark for the whole thing to go up in flames.”