Owen Paterson, the Conservative former environment secretary, has cast doubt on David Cameron’s ability to deliver the EU-wide immigration reforms he proposed in his long-awaited speech on Friday.

Paterson said that it looked as if the government had been “sat on by the Germans” and complained that relying on an EU renegotiation to address the immigration issue could take too long.

Cameron’s speech received a lukewarm reception from Eurosceptics in his party, and Paterson’s comments, on Sky News on Sunday, illustrate how strong some of their reservations are about his approach.

Paterson said that Cameron was right to say that a balance needed to be struck between allowing some workers into the UK and relieving the pressure immigration places on services, but he said that Cameron also had to deliver on what he had proposed.

“That is where I think we possibly part company because he’s looking to negotiate within the existing arrangements and it looks as if we’ve already been sat on by the Germans,” said Paterson, referring to claims – denied by Philip Hammond, the foreign secretary – that Cameron abandoned plans to propose a quota on EU migration under pressure from the German chancellor, Angela Merkel.

Paterson, who favours withdrawal from the EU, also said that getting the rule changes that would give Britain the right to stop EU migrants claiming in-work benefits for four years, as Cameron proposed, would take too long. This was “a real pressing problem”, he said.

“We cannot go into long, rambling negotiations with the European Union, particularly if they’re not going to play ball,” he said.

In a separate interview with the BBC, Graham Brady, the chairman of the Conservative backbench 1922 committee, said that Cameron should start negotiating his proposed reforms with EU partners immediately. The proposals were very important, he said, “but clearly what we want to see is the result”.

“We want to see those very big net migration figures starting to come down and I think colleagues will be much happier when we see that happening.”

The Independent on Sunday revealed that Sir Bill Cash, another prominent Conservative Eurosceptic, told a Bruges Group conference recently that more than 200 Conservative MPs agreed with him in wanting Britain to give up full membership of the EU in favour of a free trade agreement.

“We’ve moved the whole argument on the European issue as a result of both analysis and political will from ... when we were in the complete minority to a position where, although you may dispute the exact figure ... something over 200 Conservative MPs generally agree with what I’m saying today,” he said.

“We have been talking about immigration as if it is some kind of disease that needs to be treated, rather than a fact of life that has to be coped with,” he told Progress magazine. There is an important difference between the rules around which a more global world operates, and trying to opt out of it. The thing for a progressive, centre-left party not to do is to cross the line into trying to opt out of these changes.”

• This article was amended on 1 December 2014 to correct the spelling of free trade agreement.