The justice secretary's decision to cut legal aid for prisoners is a matter of political judgment, even if inmates "suffer serious adverse effects", the high court has ruled.

Dismissing an appeal by the Howard League for Penal Reform and the Prisoners' Advice Service, judges said they could not overturn decisions made by Chris Grayling, who is also lord chancellor, as part of his efforts to save £220m a year from criminal legal aid.

Lady Justice Rafferty and Mr Justice Cranston, sitting in London, said: "We can well understand the concerns ventilated through these claims. A range of impressive commentators have argued that the changes to criminal legal aid for prison law … will have serious adverse effects for prisoners. But we simply cannot see, at least at this point in time, how these concerns can arguably constitute unlawful action by the lord chancellor. For the time being the forum for advancing these concerns remains the political."

Both charities said they would appeal against the judgment. Their lawyers argued at a hearing in London earlier this month that the "unfair, irrational and inflexible" changes would undermine prisoners' rights and their chances of rehabilitation, as well as cost the taxpayer additional millions because they would trigger "hidden costs".

Phillippa Kaufmann QC, appearing for the charities, said the changes would lead to "huge unfairness" as inmates would no longer have access to legally aided advice and assistance at hearings within the prison system.

She said the cuts would stop legal assistance for female prisoners facing reviews over eligibility for mother-and-baby units, and representation and advice for inmates facing segregation and placement in close supervision units. Others affected would include Category A prisoners wanting to progress to lower categories and eventual release, as well as those with issues related to resettlement on leaving prison.

But James Eadie QC, appearing for the justice secretary, had insisted that the arguments about ministers' "victimising and targeting of prisoners" had already been considered by parliament. The new regulations were approved against a backdrop of the need for "financial stringency in the legal aid system because of scarce resources".

In the judgment, Cranston dismissed concerns that access to justice was being denied. "The [European court of human rights] jurisprudence is clear that the provision of legal aid of this character is not mandatory, except in exceptional cases … In any event the context of the present claim is not in relation to access to the courts but to proceedings before bodies such as the parole board, prison disciplinary hearings and the independent reviewer for access to mother-and-baby units.

"Legal aid for judicial review remains. We do not see that it can be argued that the removal from scope of these aspects of prison law will lead to an unlawful interference with prisoners' rights of access to the courts."

Frances Crook, chief executive of the Howard League, said: "Our legal team represents children and young people in prison. These cuts will not result in savings for the taxpayer. On the contrary, they will result in increased costs as children remain in prison for longer than is necessary for want of a safe home to go to.

"We will take this to the court of appeal as the high court made fundamental errors in its understanding of some of the key points. The court completely failed to address how unfairness would not arise in particular situations where prisoners are unrepresented. These include parole board hearings where secret evidence is used against the prisoner or other cases which turn on expert evidence that cannot be commissioned without legal representation and funding."

Deborah Russo, joint managing solicitor at the Prisoners' Advice Service, said: "We are deeply disappointed with this judgment, which fails to respond to the increased unfairness prisoners now face as a result of the latest round of legal aid cuts.

"The court is right to say that this is a political issue; however that does not mean that it is one in which the law cannot intervene if prisoners' fundamental rights of access to legal remedies are being breached. We intend to appeal the judgment and will continue to press for these cuts to be reversed and for prisoners to be provided with adequate advice and representation to defend their legal rights."