The Treasury warned Iceland it would pursue compensation through the courts to replace the cash used by the UK to guarantee savings deposits in the aftermath of the collapse of Icesave and its parent, Landsbanki.

Referendum results showed the Icelandic government had failed to clinch a "yes" vote and clear the way for an agreed repayment plan with the Dutch and British authorities. Final results showed the "no" side had just under 60% of the votes and the "yes" side about 40%.

Iceland's prime minister played down the significance of the vote, claiming the assets of Landsbanki would be recovered over several years and meet almost 90% of the shortfall. Johanna Sigurdardottir said that the vote was disappointing but would not prevent the country from pursuing its existing repayment timetable and membership of the EU.

The UK and Dutch governments want a quick return of the cash they spent replacing the Icesave money lost in the crash. The treasury minister Danny Alexander said: "We have an obligation to get that money back, and we will continue to pursue that until we do."

He told the BBC: "We have a very, very difficult financial position as a country. This money, of course, would help."

The Dutch finance minister, Jan Kees de Jager, said that the referendum result was "not good for Iceland and also not good for the Netherlands".

"The time for negotiations has passed," he said. "Iceland still has the obligation to pay us back. This is now a case for the courts."

The result reflects Icelanders' anger at having to pay for the excesses of their bankers, and complicates the country's recovery from economic meltdown. It is the second time voters have defeated an attempt to settle the bitter dispute, which stems from the collapse of Iceland's high-flying banking sector in 2008. The government said it would be the last.

"We are at the end of the road of a negotiated solution," said the finance minister, Steingrimur Sigfusson.

He said Iceland would now opt for "plan B", with the dispute going to the European Free Trade Association court, which could impose harsher terms on Iceland than those rejected in Saturday's vote.

A tiny North Atlantic nation with a population of only 320,000, Iceland went from economic wunderkind to financial basket case almost overnight when the credit crunch took hold. Its major banks, which had expanded to dwarf the rest of its economy during a decade-long, credit-fuelled boom, collapsed within a week in October 2008. Its currency plummeted and protests toppled the government.

The savings of Icelandic citizens were protected by an unlimited domestic deposit guarantee, but no such rule applied to foreigners attracted to Icelandic banks by their high-interest accounts.

About 340,000 British and Dutch savers had deposited more than £4bn in Icesave. After Icesave collapsed, British and Dutch authorities borrowed money to compensate their citizens, then turned to Iceland for repayment.

The dispute has grown acrimonious, with Britain and the Netherlands threatening to block Iceland's bid to join the EU unless it is resolved.

Failure to approve a deal also stalled payouts from a £3.8bn loan from the International Monetary Fund.

In the wake of the referendum defeat, Sigfusson said that the government would hold talks with those who had loaned Iceland money, the IMF, the Nordic nations and Poland.

"We have made substantial progress moving out of the crisis in 2008, and we intend to keep on doing so despite this outcome," he said.

Icelanders overwhelmingly rejected a previous deal in a referendum last year, but the government had hoped a new agreement on better terms would win approval.