Along with many other members of his party, Jon Rogers watched the footage of his leader's apology over the Lib Dem U-turn on tuition fees last week in a mood of apprehension, fascination and mild irritation. "Was the apology a good idea? I don't know, time will tell. Some of us would rather just forget it," admitted the Liberal Democrat mayoral candidate in Bristol. "I enjoyed the iTunes version."

As the Lib Dems gather in Brighton for their annual conference after two and a half years in government, the central problem is not so much how to atone for alleged sins committed during the first half of this parliament as how to game-plan the countdown to the next election, when the fear is that the Lib Dems could be all but wiped from the political map. The crucial second question, of course, is who should be leading the party when that moment of truth arrives.

In Sheffield, where Nick Clegg is an MP, opinion is deeply divided about the leadership – even among Liberal Democrats. There are worries in the party that come the next election he could even lose his Sheffield Hallam seat. The leader of the party on the city council, Shaffaq Mohammed, said he believed Clegg should take the party into the next election. He believed, too, that he could and should defend his seat against the second-placed Tories and third-placed Labour so long as Lib Dems got a hearing on the doorsteps.

Mohammed's optimism runs contrary to a prevailing view that the high student population in Sheffield and the number of public sector workers in the city will do for him, firstly because of tuition fees and second because of the coalition's harsh public-sector cuts. Mohammed contests this. "If we can get behind the doors and get an audience and explain why we are in government, we can counter the hostility and argue our case. We can do it," he said. But Andrew Sangar, a Liberal Democrat councillor for 21 years, whose ward is in Clegg's constituency, is less certain. While he admires Clegg and thinks he should lead the party in the short term – a period in which he believes it must make clearer its "distinctiveness" from the Tories – he is in two minds as to whether Clegg should remain in charge up to polling day. "I think it is too early to say," he said.

Among students, too, opinion is divided. A tour of university areas found some were hostile but a surprising number were getting used to the idea of student fees and suggested they had better things to do than hold grudges against Clegg.

Without doubt, party members across the country are wrestling with the Clegg question – and wondering about Vince Cable. Gary Hopkins, a cabinet member on Bristol council, would not be drawn on his leader's future but was happy to talk of the business secretary's successes. "Vince is obviously a star," Hopkins said. "Brilliant man. He is extremely highly regarded by Liberal Democrats and a lot of other people who are not Liberal Democrats."

Some Lib Dems address the leadership issue straight on, others do not. Annette Brooke, the Liberal Democrat MP for Mid Dorset and North Poole, and a renowned loyalist, is surprisingly one who does. She said there was a need for the party's identity to be asserted, and spoke of it being "healthy" that there were other leadership options. She confessed to having been "pulled in all directions" by the government's tax and welfare policies and her own party's U-turn on tuition fees. "I am very keen that this conference is laying down the Liberal Democrat policy lines as we go into the next general election. That's really important," she said.

"I am still told that when we went into the coalition it was the right thing to do. I still think it is an important thing, but there are some issues that I have found hard to deal with. I've been pulled in all directions. I think it is a balancing act but Liberal Democrats need to be clear about the party's separate identity. It needs to be really clear to the public.

"I particularly didn't support the 50p tax rate cut. It seemed totally the wrong message to send to people. We do need to be very clear that we are the party that stands for fairness. That is going to be a theme through the conference."

Asked if Clegg would continue to be leader after 2015, she too preferred to hedge her bets: "I think you never know how things will turn out. I think it is always healthy that there are possibilities. The press think that Vince will be interested in the job but we have to be focused on what we believe in."

The damning assessment of the government's borrowing policy by Richard Reeves, reported elsewhere in today's Observer, will do little to help the Lib Dem leader, who has had an uncomfortable week. Until July, Reeves was the deputy prime minister's director of strategy and his doubts about the coalition's economic policy do not exactly inspire confidence that everyone is rock solid behind it in Downing Street.

His comments were seized upon by Labour last night, who said the government was paying a "heavy price" for Clegg failing to listen to his aide's concerns while in Downing Street. Writing in a pamphlet to be launched today, Reeves insisted that the broad thrust of the deficit reduction plans is right. But he spoke of his frustration at the mistakes the government had made in what he concedes could be viewed as a "searing critique".

Reeves wrote: "Nobody knows for sure whether tightening at the pace set by the coalition government has choked off growth or laid the foundations for recovery. For what it is worth, I think the coalition tightened a little more than necessary in the first two years; relied a bit too much on spending cuts rather than tax rises to fill the hole; and above all has taken a myopically conservative approach to borrowing for investment."

The ringing phrases amount to a repudiation of much of the platform that has cost the Lib Dems popularity across the country during two and a half traumatic years. They will strike a chord with many members of the party, who have sat on their hands as the leadership has refused to budge from the austerity line that has defined the government.

Research by market research bureau TNS BMRB, seen by the Observer, shows evidence of growing concerns that Clegg has failed to influence the chancellor's economic policy, with over a quarter (27%) of Liberal Democrat supporters telling pollsters that their leader has performed very badly or fairly badly. The two top priorities for those Lib Dem voters, according to the research from a survey of 2,500 people, are economic growth and reducing the national debt.

Yesterday, as delegates gathered in Brighton, Clegg appealed to members to unite behind him and, in a sign that he understood the seriousness of the concerns, set out taxation as well as growth as an area in which his party would make the biggest difference over the next two and a half years.

In a crowd-pleasing move, aides briefed that the deputy prime minister would demand a mansion tax if Osborne attempted to find further cuts, including the extra £10bn the chancellor is said to be seeking from welfare.

Unveiling the conference slogan as "Fairer tax in tough times", Clegg told the conference: "It's just wrong that people on low and middle incomes who work hard and play by the rules are taxed so much, while Russian oligarchs pay the same council tax as some of you do on a family home. Liberal Democrats are fighting to change that. Lower taxes on work and more on unearned wealth. I want to reward people who put in a proper shift, not those who sit on a fortune. People for whom a bonus means a few extra quid at Christmas, not a million-pound windfall."

Party president Tim Farron and Liberal Democrat home office minister Jeremy Browne also told members that the coalition needed to last until 2015, and offered support to Clegg. Farron called for his ministers to be "distinctive but not destructive". Browne added that the party needed to show that it could be a "party of government".

Out in the country, however, it is not difficult to unearth dissent. John Kiely, the longest standing Lib Dem councillor in Bristol, where the party is in minority, said he would only be content at the end of conference if Clegg resigned during his keynote speech and passed the mantle to Cable. "It's not working," he said.

Peter Child, 67, a party activist from Eastleigh, in Hampshire, where Chris Huhne is the MP, added: "Clegg has not had a good time and I don't see any signs that are going to improve. He will struggle to lead the party up to 2015."

Kevin House, leader of Eastleigh council, while backing his party leader, said: "He needs to show that he isn't scared of his own shadow."

The Observer understands that at a recent meeting of Liberal Democrats in Bristol, there was resistance to inviting Clegg to stand alongside their candidate, such is the state of his poll numbers, but Jon Rogers said the leader of his party would be coming to his city. "There is no way I can pretend that I am not a Liberal Democrat. I think Nick Clegg is doing a very good job in a very difficult situation."

By and large, despite the rumblings, that is still the majority view among Lib Dems. Back in Sheffield, Andrew Sangar says that despite all the criticism, Clegg is still very well liked locally. "People are still very pleased to see him. He is still very well received."