A Manhattan judge removed herself from a case brought by two of Vito Lopez’s sexual harassment victims against the Assembly and Sheldon Silver because she is a former colleague of Silver’s attorney.

Justice Joan Madden disclosed in Manhattan Supreme Court on Tuesday that she had served on the bench at the same time as the speaker’s in-house counsel and right-hand man Jim Yates.

Former employees Victoria Burhans, 27, and Chloe Rivera, 25, sued in June claiming that Lopez would not have been able to abuse the young women “without the assistance of the Speaker of the Assembly, Sheldon Silver,” because he knew other female staff had complained about the disgraced lawmaker’s behavior.

Silver allegedly “aided and abetted” the sexual discrimination by allowing a hostile work environment, the suit says.

The women’s attorney, Kevin Mintzer, told the judge that Yates was intimately involved in the decision to sweep the seamy allegations under the rug and craft a confidential settlement.

Both Judge Madden and Yates started their careers as Legal Aid lawyers in 1973.

Madden has served as a Supreme Court judge for over 14 years.

Yates was appointed to the Supreme Court around the same time in 1998. He stepped down and joined Silver in 2011.

There is a parallel case pending in Manhattan Federal Court. The judge in that suit is considering a motion to toss the case.