Alex Salmond had confidently predicted victory on the eve of the byelection, and his deputy, Nicola Sturgeon, admitted today the result was a "very disappointing" one for the SNP.

SNP officials, habitually cheerful and positive, were close to tears as the results came in, and the shocking scale of Labour's victory became clear.

Speaking on BBC Radio 4's Today programme, Sturgeon claimed Labour had won by running a "relentlessly negative" campaign, focusing on a single local issue – a steep rise in care charges for some Fife residents by an SNP-led council.

She alleged Labour had deliberately misled voters. "We ran a very good campaign, we fought hard and had an excellent candidate, and we are very disappointed not to have won the seat," she said.

"But let's look at the facts. We substantially cut Labour's majority, our vote went up by 13% since the vote in the last general election and we recorded a 5% swing against Labour so yes, it is disappointing because we fight every election to win, but there was substantial progress for the SNP in this seat last night."

But senior SNP officials, who initially believed they had snatched the seat by up to 1,000 votes, admitted early this morning they had been out-gunned and out-thought by Labour. Some admitted they were shocked, even confused, about the scale of Labour's victory. It defied the bookies, Labour's standing in the polls, the SNP grip on power in Fife, and SNP's victory in Glasgow East in July.

Peter Murrell, the party's chief executive and architect of the SNP's remarkable climb to dominance in Scotland over the past two years, said Labour had focused its best tacticians on the campaign, but not from within Scotland, but across the UK.

"The sharpness of some of the negative material shows there was some serious brains brought in by Labour," he said.

But Labour feels this focus on negative campaigning misrepresents its campaign and downplays the role played by a new generation of Scottish Labour politicians instrumental in crafting their campaign.

One was Gordon Banks, the MP for Ochil and South Perthshire – a man to whom Labour's victorious candidate, Lindsay Roy, paid direct tribute from the election podium.

He is regarded within the Scottish party as an expert campaigner in marginals and difficult seats.

Banks took his seat, slightly altered by boundary changes, from the SNP in 2005 – the first Labour MP for that area of Perthshire, once a Tory fiefdom in Scotland.

Another prominent local figure was John Park, the MSP for Mid Scotland and Fife, is a Fifer who first entered Holyrood last year and is touted as a possible party leader in Scotland.

Park knows his county – a former trade union organiser and manager at Rosyth dockyard, one of Fife's larger employers.

Park said yesterday afternoon – not knowing then that Labour was about to retain the seat by a comfortable margin – the party had "deployed some techniques we haven't used in so complete a way before in a byelection."

Even so, Labour did face genuine anger about the aggression in their controversial campaign over Fife council's rise in some care charges, from £4 a week to £11 a week for a minority of pensioners and disabled residents.

One leaflet, based on the true story of a disabled resident sent a debtors letter by the council, run by the SNP candidate Peter Grant, had frightened many voters. Roy insisted the story was accurate – it was, but it was leveraged to its absolute limit.

These attacks on the SNP's record – in Fife and in Edinburgh – were relentless and very well funded. Murrell is right: Labour deployed money and talent not available in Scotland. If fought Glenrothes like a general election.

Yet the appearance of Sarah Brown, at least seven times, belies the negative campaigning accusation. After an awkward first brush with the media, she focused on door-to-door chats and shopping centre visits.

Then came the glossy eight-page mini-magazine sent to every voters' home profiling Roy, complete with photos of his father, an RAF pilot, and mother, a wartime nurse. Then came endorsements from Sir Alex Ferguson, the manager of Manchester United, and Ian Rankin, the author and native of Cardenden.

And Salmond and Grant faced – for the first time – the difficulties of encumbency. Both found themselves defending SNP policies at a local and national level: care charges, education policy, local income tax and economic policy came under Labour fire.

In parallel came the "Brown bounce": buoyed by his resurgence with the financial crisis, his personal standing and prestige in Fife may turn out to be the deciding factor.

Perhaps too, the SNP will decide its campaign was too narrow: focusing very heavily on fuel costs, fuel poverty and rising prices.

The key point on the day was that Labour mobilised far more voters than either Labour or the SNP expected.

Tricia Marwick, the SNP MSP who took the equivalent Holyrood seat from Labour last year, said the nationalists believed they could win the seat with 14,000 votes.

They nearly achieved that, gaining 13,209. Yet Labour turned out 6,000 more voters than expected – defying all predictions and the poor winter weather. The turnout was a very respectable 52.3%, only a few points down on the general election.

"Our SNP vote came out today, and I saw people coming out to vote for Labour who haven't voted for 20 years. I need to think about that," Marwick said.