The High Court has rejected a challenge by charities working with prisoners over legal aid cuts introduced by Justice Secretary Chris Grayling.

The Howard League for Penal Reform and the Prisoners' Advice Service (PAS) say vulnerable people in the prison system, including inmates with mental health problems and mothers with young babies, will suffer injustice following the removal of the right to criminal legal aid in many prison law cases.

Lady Justice Rafferty and Mr Justice Cranston, sitting in London, said they could "well understand the concerns" raised by the new regulations, introduced last December by Mr Grayling, who is also the Lord Chancellor.

They added: "But we simply cannot see, at least at this point in time, how these concerns can arguably constitute unlawful action by the Lord Chancellor.

"For the time being, the forum for advancing these concerns remains the political."

The Howard League and PAS accuse Mr Grayling of imposing the cuts in England and Wales for "ideological reasons entirely contrary to the rule of law" on inmates who will be "too vulnerable to complain".

Their lawyers argued at a hearing in London earlier this month that the "unfair, irrational and inflexible" changes will undermine prisoners' rights and their chances of rehabilitation, as well as cost the taxpayer additional millions because they will trigger "hidden costs".

Phillippa Kaufmann QC, appearing for both charities, said the changes will lead to "huge unfairness" as inmates no longer have access to legally-aided advice and assistance at hearings within the prison system.

Ms Kaufmann said the cuts stop legal assistance for female prisoners facing reviews over eligibility for mother-and-baby units, and representation and advice for inmates facing segregation and placement in close supervision units.

Others affected include Category A prisoners wanting to progress to lower categories and eventual release, as well as those with issues related to resettlement on leaving prison.

The QC also contended that legal aid has been unlawfully removed from prisoners' hearings involving the Parole Board where the board does not directly have the power to direct a prisoner's release.

Ms Kaufmann said: "People are not going to be able to adequately represent themselves. The cuts will also interfere with the right of access of prisoners to the courts.

"The Secretary of State has made absolutely clear he doesn't believe that prisoners should have access to legal aid to go to court about these matters.

"These are ideological reasons entirely contrary to the rule of law. He has created a system that is inherently unfair."

James Eadie QC, appearing for the Justice Secretary, said all the arguments now being put in court about ministers "victimising and targeting of prisoners" had already been considered by Parliament, but the new regulations were approved against a backdrop of the need for "financial stringency in the legal aid system because of scarce resources".

Mr Eadie said the "punchline" that the cuts were unfair did not work because Ms Kaufmann could not point to any lead cases of unfairness having actually arisen.

If there was unfairness at internal hearings or reviews in the prison justice system, or before parole boards, civil legal aid was available for complainants to use the safety net of seeking judicial review.

If prisoners needed extra help because of vulnerability or mental health issues, "no doubt proper allowance will be made for that", said Mr Eadie.

He said the UK legal aid system "is pretty much at the top of the tree in generosity" compared with the rest of Europe, but it was now necessary to get the multimillion-pound budget under control.

In reply, Ms Kaufmann said judicial review was no answer because the changes in the system meant that, now prisoners were denied legal assistance, "no one will know" if they are entitled to bring claims against the prison authorities.

She said: "These victims of unfair decisions will not know where to go. They don't have any lawyers any more to help them."

But today Lady Justice Rafferty and Mr Justice Cranston refused to allow the legal challenge to go forward to a full judicial review, despite the concerns raised.

They ruled the case raised issues "of political judgment and prediction into which the courts cannot venture".

Both the Howard League and PAS said they will appeal against today's ruling, accusing the High Court of making "fundamental errors".

The Howard League's chief executive, Frances Crook, said: "Our legal team represents children and young people in prison.

"These cuts will not result in savings for the taxpayer. On the contrary, they will result in increased costs as children remain in prison for longer than is necessary for want of a safe home to go to.

"We will take this to the Court of Appeal as the High Court made fundamental errors in its understanding of some of the key points.

"It did not properly deal with the concerns of the Joint Committee on Human Rights that the complaints system cannot be effective in certain cases.

"The court completely failed to address how unfairness would not arise in particular situations where prisoners are unrepresented.

"These include parole board hearings where secret evidence is used against the prisoner or other cases which turn on expert evidence that cannot be commissioned without legal representation and funding."

"We run an advice line and receive thousands of letters and telephone calls from prisoners each year. PAS also represents prisoners by taking on legal cases where appropriate.

"We are deeply disappointed with this judgment, which fails to respond to the increased unfairness prisoners now face as a result of the latest round of legal aid cuts.

"The court is right to say that this is a political issue; however, that does not mean that it is one in which the law cannot intervene if prisoners' fundamental rights of access to legal remedies are being breached.

"We intend to appeal the judgment and will continue to press for these cuts to be reversed and for prisoners to be provided with adequate advice and representation to defend their legal rights."

Legal Aid minister Shailesh Vara welcomed today's ruling, saying: "Legal aid should be used for addressing genuine injustices - not as an issue for campaigning by pressure groups or for minor complaints from prisoners that can be fairly dealt with by other means.

"We welcome this judgment, which rejects the notion these changes are unfair or unlawful.

"It is regrettable we had to spend more taxpayers' money making these points in court."