Relations between Rupert Murdoch's News International and the government disintegrated today during 24 hours of recriminations over the move by its flagship paper, the Sun, to switch allegiance to the Conservatives.

Gordon Brown moved quickly to deny any political damage, saying: "It is the people who decide elections." But on one occasion the prime minister was clearly tested, tearing at his ear piece when he made an irritated and overly hasty departure from one TV interview, prompting aides to clarify later it had not been an angry "walkout". Senior party figures were buoyed by new YouGov polling that showed support for Labour had gone up. Conducted after Brown's setpiece speech and before the Sun's pronouncement, the party climbed by six points within five days, halving the Conservatives' lead, according to the polling

Throughout the day in Brighton, senior figures made colourful put downs of the Sun's switch. Tony Woodley, joint leader of the Unite union, was cheered as he used a speech to the conference hall to rip up a copy of that morning's edition, while Labour's deputy leader Harriet Harman told the conference hall Labour would not be "bullied" by the paper.

Lord Mandelson, the business secretary, told News International's chief executive, Rebekah Brooks, on the phone on Tuesday night: "You will be regarded as a bunch of chumps, we will not lose sleep over this." Earlier reports said Mandelson had used an expletive beginning with C, but Brooks is said to have rung the business secretary to confirm he called her a "chump".

According to sources at News International, Brooks tried repeatedly to make calls to Brown on Tuesday night to warn of their new editorial line but the prime minister was tied up at an evening reception. Instead Brooks contacted Mandelson to tell him about the Sun's front page – headlined "Labour's lost it" – and the business secretary replied that the party would not be "terribly surprised".

Labour sources were bullish this afternoon as Downing Street said the polling showed they were "grinding down" support for the BNP and Ukip votes, something they regard to be key to standing a chance at the next general election. Mandelson welcomed it cautiously, saying "one swallow does not a summer make".

Labour MPs were quick to dissect the likely impact of the Sun's decision on the polls. Liverpudlian Woodley's ridicule of the paper was a function of its limited popularity in the north-west after its coverage of the Hillsborough disaster.

MPs in key marginals report the Mirror to have greater purchase than the Sun.

Ministers believe the timing of the new editorial line is not a positive endorsement of David Cameron, or it would have been made after the Tory leader's own speech.

One senior Labour source said: "The Sun is trying to beef up its declining circulation. It vastly overestimates its influence."

The source also compared George Pascoe-Watson, the paper's political editor, and Trevor Kavanagh, its associate editor, to old-fashioned union leaders, criticising them for "strutting around the Labour conference like old union barons with their block votes of 10m readers".

David Cameron told LBC radio he was "delighted" by the switch. But former deputy prime minister John Prescott said via Twitter: "It will be the Son, Daughter, Uncle, Mother and Friend Wot Win it in 2010. Endorsements from ordinary people NOT media barons."

The Sun's decision to withdraw its support from Labour just hours after the prime minister's speech raises questions about the media's influence on British politics. By James Robinson and Stephen Brook