Labour leader Ed Miliband meets members of the public in Manchester. There are concerns he is unable to connect with voters. Photograph: Nigel Roddis/Getty Images

Ed Miliband is facing a crisis of confidence among his backbenchers, with direct warnings being sent to the leader’s office that he has to improve his performance and complaints being channelled through the chair of the parliamentary Labour party, David Watts.

The news came as Andrew Harrop, the general secretary of the Fabian Society thinktank which is affiliated with Labour, admitted Miliband was a hindrance to the party’s success, and claimed Miliband’s own confidence had been badly hit. He said, however, that he had not noticed a groundswell of members demanding that Miliband stand down.

But speaking on a visit to Northamptonshire, Miliband dismissed rumours that there was a backbench move to oust him as nonsense, insisting the party was united in its effort to set out policies to build a new economy.

“I don’t accept that this matter arises,” he told the BBC.

“ This is nonsense. My focus, and the Labour party’s focus, is on the country, and the things that matter to the country. That’s the cost of living crisis, the NHS, it’s the prospects for the next generation. That’s my focus here in Northampton and that’s our focus across the country ...

“There are huge issues that the country faces, issues of why the country doesn’t work for most people. That is what we are determined to change. We are determined to be a one-term opposition that changes that.”

The rapid decline in confidence stems from a party conference that left members depressed and saw Labour’s poll rating slip close to the levels endured by Gordon Brown in the 2010 election. Miliband appears to have been unable to regather momentum since the conference.

Watts has been asked to pass complaints about Miliband’s leadership directly to him. The Labour leader has responded by organising a shadow cabinet reshuffle, bringing in two of his most trusted allies, Lucy Powell and Jon Trickett, who were key figures in his leadership campaign. Powell has said she will unblock the processes which prevent candidates, frontbenchers and Miliband from shining.

The reshuffle prompted anger among those who believed loyalty had not been rewarded in the shadow cabinet.

The crisis comes after four years of unity across the party, but a growing fear that Miliband does not have the ability to connect with voters.

Miliband’s office denied that he would issue a statement on Thursday to address doubts about his leadership.

Meanwhile, party officials denied reports that the reshuffle undertaken on Wednesday had been prompted by the possibility that he was about to receive a letter from backbenchers calling on him to stand down. They said the allegation that such a letter existed was put to the party after the reshuffle, and added that the leader’s office was not aware of any such letter.

A Labour spokesman said: “These are extremely febrile times in British politics. No one can know quite where we are going to end up. But there is one party in British politics that is panicking, taking risks with the national interest and making promises it knows it cannot keep, and that is the Conservative party. The only party that is setting out clear and concrete policies to improve the British economy that is not floundering around that is not threatening to leave the European Union and not panicking and that is Labour.”

He said there were always one or two people in the parliamentary party that might be unhappy with the Labour leadership, but that was true at the time of Tony Blair in 1996 as well as now.

But senior leadership sources said they accepted there was a rebellious mood in the party and deep frustration that Labour seems to be being squeezed by Ukip in England and the Scottish National party in Scotland. There is also anger at the slowness with which decisions are being made, Miliband’s sporadic media appearances and the failure to give shadow cabinet members and rising stars a higher profile.

Underlying all this is a fear that Labour’s already declining support could be hit even further during the election campaign.

Ed Balls, the shadow chancellor, sprang to Miliband’s defence: “I’ve no idea about any of this. All I know is that everybody in the Labour party, from Ed Miliband down, is focused on tackling the cost-of-living crisis, building an economy which works for working people, reforming Europe but not walking away, getting tough and fair controls on immigration, saving our National Health Service – that’s what Labour’s for. It’s the Conservative party which are riven and divided and defecting, left, right and centre. Labour will stay united.”

Harrop said: “I have not detected any groundswell of support for a leadership change and there is no alternative candidate. There is also the sense that spending three months with the party discussing its own leadership rather than getting on and fighting the election would be a colossal waste of time, which might not lead the party forward in any way, further than it is now … The party needs to collectively get its confidence together and back him, and Ed needs to up his game as well.”

But he conceded: “At the moment Ed is a hindrance to the Labour party’s success. But people are still saying they prefer a Labour government to a Conservative government in the polls and they know that Ed Miliband is the leader.

“We have looked at how Labour’s polling has fallen in the last year or so and Labour does have serious problems – it has lost many of its own 2010 voters who supported Gordon Brown and are now wondering whether to support Labour again. It has to get those sorts of people back, but changing leader at this stage is not going to help in that job.”

Brown’s former adviser Damian McBride said Labour was whistling in the wind if it thought it could make Miliband more popular by putting him out on the doorstep.

Asked if he gave credence to claims about a letter being circulated among backbenchers, he replied: “I don’t know, and it’s difficult to know because the paranoia that comes out of the Miliband camp is so rank that they will invent plots even when there are none.

“But I think the mood is pretty black in Labour, and certainly since the conference. Since party conference the mood has got blacker and these are wild times.”