George Osborne at last year's Conservative party conference. Photograph: Andrew Yates/AFP/Getty Images

Just after becoming prime minister, David Cameron justified the wide-ranging impact of spending cuts by saying the government could not cut the deficit "by just hitting either the rich or the welfare scrounger".At last year's party conference the chancellor, George Osborne, said: "Where is the fairness, we ask, for the shift-worker, leaving home in the dark hours of the early morning, who looks up at the closed blinds of their next-door neighbour sleeping off a life on benefits."

And since then Conservative ministers have repeatedly implied disdain, or at least a lower priority, for non-workers with references to policies to help "strivers" and "hard-working families".

The welfare secretary, Iain Duncan Smith, is said to regularly disown such comparisons, stressing it is "not my language". On Tuesday the former coalition Liberal Democrat minister Sarah Teather warned MPs the labels would "set neighbour against neighbour".

Then Tory officials insisted the focus on language was a distraction from their challenge to Labour about how the opposition could support a 1% cap on public sector pay but not on benefits.

The Treasury minister Sajid Javid upped the stakes, quoting back at Labour the shadow work secretary Liam Byrne's conference speech two years ago, when he said: "Let's face the tough truth – that many people on the doorstep at the last election felt that too often we were for shirkers, not workers."

Politicians on all sides are steered by polls suggesting low public support for raising benefits, especially for the unemployed (about 15% believe they should get more). Whether insulting them with such sweeping stereotypes at the same time helps win or lose votes has yet to be seen.