Nick Clegg has expressed his regret that Lord Rennard will not apologise to women who claim he harassed them, but will not try to force the peer out of the Liberal Democrats.

The deputy prime minister admitted his own failure to listen to the women's initial allegations against the former Lib Dem chief executive, but said there was little more he could do after an independent inquiry found there was not enough evidence for disciplinary action.

The investigation by Alistair Webster QC said Rennard ought to apologise and commit to changing his behaviour, which may have caused "distress" to women by violating their personal space. However, it found there was not sufficient evidence that Rennard had acted in a sexually inappropriate way to bring any disciplinary charges.

Some of the Lib Dem women who accused Rennard of harassment have criticised their party for "cowardice" in failing to take action, allowing the peer to return to his job of helping to write Clegg's 2015 election manifesto.

He said it was a "terrible example of secret justice" and astonishing that neither he nor Rennard had been allowed to see the independent report.

Speaking on the BBC Radio 4 Today programme, Carlile said Rennard would not be apologising since he had done nothing wrong, adding that he would be resuming his position on the Lib Dem federal policy committee.

He also insisted that Webster had told him and Rennard directly on the phone that the case against Rennard did not meet either the criminal or the civil standard of proof.

Rennard's refusal to apologise leaves Clegg in a difficult position, as he has promised the peer will have no role in his election campaign but can do nothing to stop him returning to a senior role in the party.

Speaking on his LBC 97.4 radio show, Clegg said: "He should apologise and it's not me just saying that. The QC, the independent QC, an eminent judge, looked into this. It was done entirely independently and … he said there wasn't in his view enough evidence to clear the threshold of what is in effect the criminal burden of proof in order for him to recommend action.

"But he felt there was, in his view, clear evidence that the women were speaking with credibility, that they should be believed, they were subject to behaviour that was distressing, that Chris Rennard should reflect on his behaviour and he should apologise and I think it is a matter of very real regret, to put it mildly, that, so far at least, he hasn't chosen to apologise.

"I want him to do the decent thing as asked for by the QC … who said: 'Look, you just owe an apology.'"

Clegg also spoke of his own failures and said he would look at improving Lib Dem disciplinary rules.

"The buck stops with me and I've said very clearly there are a number of failures … I think the party as a whole and myself as leader, we've got to take responsibility for the fact we failed to respond to the concerns first raised."

Two of the women who complained about Rennard expressed their disappointment on Wednesday. Alison Goldsworthy, a Welsh activist who spoke out on Channel 4 News last year, accused the party of cowardice and said she was taking legal advice.

"They have failed to say Lord Rennard's behaviour is unacceptable, they have failed to discipline him and therefore have failed to give victims the justice they deserve," she said.

"Since this news became public, people from across the political spectrum have approached me to say they too had been a victim of someone abusing a position of power. Today's verdict, and failure to take action, sends the message that this behaviour can go unpunished.

"Now more than ever, the political parties must match their rhetoric with action. Women should not be afraid to speak up, and good men should not be afraid to support them. I stand by my previous statement and am taking legal advice."

Alison Smith, now an Oxford University politics tutor, who also complained about his behaviour, said Rennard should "absolutely not" return to the federal policy committee and the party needed "new procedures, and fast".

She said: "The Liberal Democrats have much soul-searching to do. They orchestrated a cover-up for many years to avoid dealing with the serious and numerous allegations against Lord Rennard."

Bridget Harris said she would tear up her Lib Dem membership card.

She told Channel 4 News: "They know what happened, we know we're not lying, and so we've ended up with a situation where the party has said it is acceptable for somebody like Lord Rennard to have done the things that we are claiming that he has done. It's acceptable to still have him as a member of the party.

"For me that's not good enough. And it's not good enough for me to carry on being a member of that party."

Harris later said the deputy prime minister had apologised to her and he was "flabbergasted" by the situation. She told BBC2's Newsnight: "I spoke to Nick earlier on this evening personally. He called me and apologised and I take that completely sincerely."

Comments issued by Rennard made no mention of any apology to the women and emphasised that this was now the end of all investigations into allegations about his behaviour, as the police dropped their inquiries last year.

"My wife, Ann, and I are very grateful to our families and many friends both inside and outside the Liberal Democrats who have shown unwavering support throughout," he said.

"I now look forward to resuming my roles within the Liberal Democrats. I remain committed to the same values which made me join the Liberal party as a teenager more than 40 years ago."