He is one of several party figures - including former chairman Lord Saatchi and ex-treasurer Lord Kalms - who have criticised the Tory leader since the party came third in two by-elections on 19 July.

Mr Cameron's decision to visit Rwanda while parts of England were badly flooded has also come under fire.

Mr Cameron has rejected the criticism saying he is addressing the "big issues" which matter to voters.

"What matters is actually delivering for the British people a really compelling alternative that actually meets the things they care about," he said.

He said Labour - under the new Prime Minister Gordon Brown - could not make the changes necessary to mend Britain's "broken society" and was too "wedded to state control".

Criticism 'inevitable'

In a speech on school discipline earlier he pledged to scrap local authority appeals panels which can overrule schools which exclude badly-behaved pupils, saying it undermined the authority of head teachers.

But asked about discipline within his own party, Mr Cameron said that the direction he had taken - including a refusal to promise tax cuts and to adopt "opt outs" from the NHS - had "inevitably" lead to criticism.

But he said he was determined to keep "going on the course I have set".

And he insisted Labour's performance in recent polls - which seem to suggest the "Brown bounce" is continuing following Gordon Brown's succession as prime minister in June - were temporary.

A survey for the Times put the Labour party six points clear of the Conservatives with 39% of the vote compared with 33%. Populus questioned 1,511 adults by telephone between July 27 and 29.