Westminster is slipping into a cycle of perpetual coalition, risking instability and creating "democratic deserts" in safe seats where votes don't influence the overall outcome of an election, a leading left-of-centre thinktank warns today.

The current first past the post (FPTP) system of electing MPs is "broken" and unfit for modern elections that are no longer a contest between just two main parties, according to the Institute for Public Policy Research (IPPR). With the rise of smaller parties and the increasing concentration of the Labour and Tory vote in their respective heartlands, the vote is split in a way in which coalitions are likely to become the norm, the IPPR report argues.

It is a significant intervention in the debate over electoral reform in advance of the May referendum on scrapping FPTP and replacing it with the alternative vote (AV) system. The anti-AV campaign has argued that the current system produces stronger, more stable governments than would potentially be elected under AV, in which voters can rank their candidates and the lowest scoring are eliminated until you have a candidate with at least 50% of the cumulative votes.

The report comes as MPs prepare to return to Westminster next week with a byelection in Oldham East as well as the Scottish, Welsh and local elections coinciding with the AV referendum in May, in what will form a series of tests of the coalition's popularity. Simon Hughes, the Liberal Democrat's deputy leader, yesterday categorically ruled out any form of electoral "pact" with the Tories after concerns from Conservative backbenchers that the parties are heading in that direction.

The referendum on the alternative vote is the most crucial test for the Liberal Democrats this year, having won it in the coalition deal and with elements of the "no" campaign attempting to turn it into a referendum on Nick Clegg's place in government.

But the issue has also firmly split the Labour party with big names such as David Blunkett and Margaret Beckett opposing it while the leader, Ed Miliband, is supportive. David Cameron will be in the awkward position of campaigning against changes after instigating, albeit at the Liberal Democrat's insistence, the referendum.

The IPPR report says: "UK voters are fed up with a two-party politics which FPTP is struggling to sustain but which still militates against the electorate's desire for greater pluralism. In short, Britain has over the last 30 years evolved into a multi-party system which retains an electoral system designed for only two parties."

Nick Pearce, director of IPPR, said: "Britain now has a broken voting system that needs to be fixed. Unless first past the post is reformed the UK will be left with a voting system that neither delivers fair representation nor single party government."

Matthew Elliott, campaign director of the cross-party No to AV campaign, said: "Everyone agrees that there's no such thing as a perfect voting system, but voters have a choice between just two options on 5 May.

"We can either keep our existing way of voting or switch to the Alternative Vote, a system which is obscure, unfair, and expensive – and which Nick Clegg has called 'a miserable little compromise'."

A Guardian poll before Christmas, the first to ask the precise question that will be put to the referendum, suggested that the Lib Dems could still win the vote. Hughes yesterday insisted it was still possible.

He told the BBC's Today programme: "Liberal Democrats want a fair electoral system. For many people in Britain there is no general election – the election doesn't come anywhere near them in safe seats. I just call on people in all parties and outside politics, just to think: a change that says you can choose, you can order the candidates in the order of your support, you can decide who you prefer, you don't have to vote negatively. It means that every single MP will have to have more than half of the constituents in their constituency supporting them."

He also insisted that there would be "no deal" with the Conservatives at any forth-coming election including the Oldham East and Saddleworth byelection. "We're fighting on the issues on the ground, which are jobs and manufacturing and making sure the North West becomes prospectus again. And I can see no prospect that at the next general election there will not be a full slate of Liberal Democrat candidates in every seat in the country because our members, our supporters, expect us to be a separate proud party."