A major spanner has been thrown into coalition plans for House of Lords reform after a joint select committee surprisingly agreed to recommend that an elected second chamber should only be introduced after the constitutional change was endorsed in a national referendum.

The committee is due to publish its findings at the end of the month, but neither coalition party supports a referendum on the reform.

Labour, in its election manifesto – and many Tory MPs opposed to an elected second chamber – support a referendum. It is certain they will use the recommendation as a justification to table this demand when the bill starts its parliamentary passage later this summer. The joint committee is proposing that the referendum be held once the legislation is passed.

It is recommending a 450-strong upper house in which 80% are elected and 20% appointed. They would be elected on non-renewable 15-year terms to make them independent of party whips.

Nick Clegg has set huge store by achieving Lords reform in the coming parliamentary session, and sees David Cameron's support for the measures as a touchstone of the coalition. But Clegg now faces the certainty that the demand for a referendum will become a major focus of the reform. Labour and the rebel Tories will argue that if it was necessary to hold a referendum on the proposed switch to AV to elect MPs, one should be held if a new voting system is proposed for the second chamber. Clegg and the prime minister argue that the two proposals are not comparable since the government is drawn from MPs.

Clegg has promised to take into account the joint committee's recommendations before publishing the coalition's final bill around the time of the Queen's speech.

The joint committee has also decided to back a little-used voting system for the election of peers. It is backing an "open preferential voting system", used in Australia, which allows voters to choose whether to vote for the list of candidates compiled by the parties or to choose between the candidates by ranking them in personal order of preference.

In practice in Australia, voters tend to choose to vote for the list proposed by the party. This has prompted British critics, including the Electoral Reform Society, to complain that this is in effect a way of introducing a closed-list system that makes candidates dependent on support from party headquarters to get on to their party list, so putting a premium on party loyalty. The draft bill had proposed using the single transferable vote (STV), a system based on regional districts, with no more than seven members elected in each district.

There had been an attempt to make peers more accountable to the electorate by giving them only 10-year terms, but this was not supported.

The committee, as has been previously reported, also backed a reduction in the number of bishops to 12 as part of an overall reduction in the size of the second chamber. A proposal to reduce the number to five was narrowly rejected.

The committee has not set out any specific proposals on how the powers of the Lords vis-à-vis the Commons should be codified, except to argue strongly that the powers of the Lords will change once it is granted greater legitimacy through an elected mandate. The draft bill, prepared by Clegg and his Conservative Cabinet Office colleague Mark Harper, simply assumes that the powers of the Lords will remain unchanged, and is largely silent on the issue.

The joint committee is deeply sceptical about this assertion, and suggests proposals for codification of the relative powers of the two chambers will have to be developed over time. It argues that if the bill sets out the competing powers of the two houses in statute, any decision of either house would be subject to repeated judicial review on the basis that one or other house had overstepped its powers.

Clegg has said he is open to putting in the bill the so-called Salisbury convention, which requires the Lords not to vote down proposals in a governing party's manifesto.

But it is the proposals for a referendum that will unnerve the government most.

Katie Ghose, chief executive of the Electoral Reform Society, said the committee's decision not to recommend the STV was an opportunity missed.

"If you hold the power to decide how Britain is run, you should be elected by us, the British public. That's democracy. This mangled recommendation from the committee would mean it is once again the parties who decide who gets a seat in the House of Lords," she said.

"The original system proposed for the Lords, the single transferable vote, is widely acknowledged as the best fit and is supported by learning from the last hundred years of debate. STV would help create a chamber free to provide objective scrutiny as apposed to a mini-replica of the commons, full of party-politicians with pre-set loyalties.

"The committee needs to reassess its priorities. We cannot allow the politicians to sneak party appointments in through the back door."