Three former Labour MPs and a Tory peer accused of fiddling their expenses face criminal trials after the court of appeal ruled that parliamentary privilege did not protect them from prosecution.

Lord Judge, the lord chief justice, and two other judges rejected the argument by David Chaytor, Elliot Morley, Jim Devine and Lord Hanningfield that their fate "should lie within the hands of parliament".

Judge said: "In our judgment no question of privilege arises and the ordinary process of the criminal justice system should take its normal course, unaffected by any groundless anxiety that they might constitute an infringement of the principles of parliamentary privilege." He went on: "The stark reality is that the defendants are alleged to have taken advantage of the allowances scheme designed to enable them to perform their important public duties as members of parliament to commit crimes of dishonesty to which parliamentary immunity or privilege does not, has never, and, we believe, never would attach."

The judges were told, at a hearing in June, that the challenge was not an attempt to "take them above the law", but to ensure they were adjudicated by the "correct law and the correct body". It was said by the defendants that submitting an expenses form was part of the proceedings of parliament, and therefore protected by parliamentary privilege.

The decision – by Lord Judge, master of the rolls Lord Neuberger and Sir Anthony May – upheld an earlier ruling by a judge at Southwark crown court in central London that they were not protected by parliamentary privilege.

In a judgment citing cases from as far back as 1629, Lord Judge said: "It can confidently be stated that parliamentary privilege or immunity from criminal prosecution has never, ever attached to ordinary criminal activities by members of parliament."

With the exception of the exercise of freedom of speech, he said, "it is difficult to envisage circumstances in which the performance of the core responsibilities of a member of parliament might require or permit him or her to commit crime".

The court, he added, could not "discern from principle or authority that privilege or immunity in relation to such conduct may arise merely because the allegations are based on activities which have taken place 'within the walls' of parliament".

Lord Judge said: "If the allegations are proved, and we emphasise, if they are proved, then those against whom they are proved will have committed ordinary crimes.

"Even stretching language to its limits, we are unable to envisage how dishonest claims by members of parliament for their expenses or allowances begin to involve the legislative or core functions of the relevant House, or the proper performance of their important public duties."

Former Bury North MP Chaytor, 60, of Todmorden, West Yorkshire; ex-Scunthorpe MP Morley, 58, of Winterton, north Lincolnshire; Devine, 57, of Bathgate, West Lothian, formerly MP for Livingston; and former Essex county council leader Hanningfield, who is also known as Paul White, 69, of West Hanningfield, Essex, are all on unconditional bail and due to stand separate trials at Southwark crown court.

The four men, who all deny theft by false accounting, can still take their cases to the supreme court for a further challenge.

The charges carry a maximum sentence of seven years' imprisonment.

Lord Judge also warned that while the issue of parliamentary expenses had "excited huge public interest and, certainly on occasions, profound concern", the law had yet to take its course.

"Just because of the publicity which the cases have attracted, it remains to be emphasised that none has been convicted, each is presumed to be innocent, and the evidence in support of the allegations made against them has not yet been tested, evaluated and judged by a jury," he said.