David Cameron decided Dave the Chameleon was his favourite video. Photograph: PA

Not long after David Cameron became leader of his party, Labour ran a mocking political broadcast featuring a cartoon character called Dave the chameleon. The soundtrack was provided by Culture Club singing "I'm a man with no conviction". It was a textbook case of how an attack can inadvertently help an opponent. Tory brand managers were delighted by this presentation of their man as a cute, bicycle-riding creature who could adapt himself to a rainbow of colours. This was exactly the impression he was trying to convey to woo voters who had previously rejected the Conservatives for being too harshly ideological. This was also a time when he had taken up the environment as an issue – "Vote Blue, Go Green" – to project himself as a new kind of Tory leader. The chameleon idea helped him with that too, so much so that he joked that it was his favourite video.

This was the "modernising" phase of Cameron. It was quite some time ago now and it did not even happen at all according to the Conservative party, which has just conducted a digital purge of its website. All the speeches, videos and press releases from before 2010 have been expunged. Airbrushed from Tory history has been that period when chameleon Cameron pledged to stick to Gordon Brown's spending plans, repudiated Margaret Thatcher by declaring that there was "such a thing as society", proclaimed that "hoodies" needed "more love" and cried: "Let sunshine win the day!"

As far as the Tory party is concerned, Cameron 1.0 never existed. Since it is beyond their powers to wipe our memories as well, some of us can still recall that time when his pitch was as a centrist, modern Tory who variously described himself as a liberal, compassionate or One Nation Conservative. We can also remember the electoral point of it, which was to win over voters who had been repelled by his party in the past. We can further recall, which the Tory party seems to have forgotten, that it was really rather successful for a while.

Some of it survived into government. He legislated for gay marriage against raging internal opposition, though it is a bit moot whether he would have done so had he known just how much trouble that would cause with his party. Another legacy of his modernising period, the commitment to increase overseas aid, has also been kept. He stuck to that one even when many Tories, including some of his closest allies, urged him to drop it.

More numerous are those examples of Tory modernisation that have been abandoned. He came to office on a pledge not to mess with the NHS and then allowed Andrew Lansley to unleash one of the biggest, and most expensive, upheavals in its history, a reorganisation that will be blamed, whether fairly or not, if there is a winter flu crisis. The "Big Society" shrivelled, curled up and died. In opposition, the Tory leader attacked "capitalism without a conscience" and "markets without morality" – stronger stuff than you get from Ed Miliband. Yet he now calls the Labour leader a Marxist for suggesting that energy companies are overcharging their customers. The green agenda has been impeded all along by a chancellor who never believed in it and now Mr Cameron seems prepared to chuck it aside in his panic to find a response to Labour's pledge to cap bills.

What happened? The conventional answer from Tory modernisers is one word: austerity. Being a "compassionate Conservative" was impossible to make compatible with presiding over deep spending cuts. A member of the cabinet once said to me: "The problem is that we have a sunshine prime minister in rainy days." Still, what was quite impressive about his early period in office was that he did try to reconcile the two. In his New Year message of 2011, he declared: "I didn't come into politics to make cuts", presenting himself as motivated by difficult necessity, not ideological zeal. What a contrast with his speech to the lord mayor's banquet last week in which he seemed to advocate a never-ending squeeze on public spending by arguing for "a leaner state".

For some, this tore away what remained of the compassionate Conservative mask. Or it could be another case of Mr Cameron bearing the impression of the last person who sat on him. In this case, it would be George Osborne, a more ideologically sharp character than his friend at Number 10. The chancellor wants to shift the political battle back on to the size of the national debt and away from living standards, Labour's best territory.

One view of David Cameron is that he didn't really believe a word of it in the first place. Tory modernisation was only ever a shallow rebranding exercise. It is probably closer to the mark to say that he never developed it sufficiently, nor succeeded in converting enough of his party, to make it robust when confronted by all the adversities and events that hit a leader. He recently likened being prime minister to "being in an asteroid shower, things flying at you every day". That suggests he sees the principal challenge of leadership as dodging incoming projectiles rather than steering by any particular lodestar.

Recently, I've been asking senior government figures a fairly straightforward question: what does the prime minister believe in? One supporter ventured "responsibility", a solid-sounding word and at the same time a vague one. One cabinet member smiled at my invitation to explain the prime minister's convictions before replying: "Good question." Another senior minister exploded: "I don't think he believes in bloody anything!" A third said: "The trouble with Cameron is that 90% of his ambition was fulfilled the night he walked over the threshold of Number 10."

This is a critique that is shared by observers of his premiership from the left and disappointed spectators on the right, who include some people who would claim to be his friend. He loves being prime minister, they say, but he is not that interested in changing the country. David Cameron doesn't see all that much wrong with a world in which he and people like him are in charge.

Perhaps the most plausible explanation of what makes him tick came from within Downing Street. "He sees it as his job to look in charge and respond coolly to events as they arise while gradually nudging the country in a more Tory direction."

He is not a philosopher-politician and, to be fair, has never pretended to be one. The first time I interviewed him as Tory leader, he physically recoiled when I asked him to define Cameronism.

Talking about the coalition the other day, he remarked that on the whole there was much to be said for compromise in politics and in life. That is one of his unforgivable crimes in the eyes of the more ideologically fervent members of his party who complain that he is not really a Tory at all. That is not only wrong, it also fails to appreciate the virtues of having a compromiser as a leader at this period of their history. A less flexible personality would have found it much harder to form the coalition with the Lib Dems and then to keep a double-headed government on the road for so long. In his early period at Number 10, the willingness to share power with another party in the national interest was highly attractive in the eyes of many voters. To an extent, it still is. He remains more popular than his party.

A virtue in handling coalition relations, his lack of many deep-seated convictions has been a vice when it comes to dealing with his own tribe. He has appeased rather than confronted the right of his party. They have squeezed him for concessions throughout his premiership. His most spectacular defeat at their hands was pledging a referendum on EU membership. If it ever happens, it will be one of the greatest ironies of his premiership that he will be defined by a referendum that he never wanted.

Because he didn't sufficiently develop an idea of modern Toryism for the 21st century, he defaults under pressure back to conventional rightwing positions. The Conservatives are defining themselves as the party of welfare cuts, immigration controls, deficit reduction, spending squeezes and loathing for Europe, which wasn't the original idea at all. The man who once embraced coalition with the Lib Dems as another way of demonstrating that he was a One Nation Tory now snarls at Nick Clegg for stopping him being as rightwing as he would like. Tory tribalists may like the sound of that, but it dismays those in his party who understand that Conservatism only attracts support across the country when it is broadly appealing.

One of the worried is the last Tory leader to win an election, Sir John Major. He has gone public with his fears that his party has forgotten that it needs to demonstrate that it has "a heart and a social conscience" and is retreating rightwards into a narrow Tory "comfort zone".

Labour might now have more success with Dave the chameleon than they did when it was first broadcast. Said the narrator: "Underneath it all, he was still blue, through and through."