The trustee seeking to recover money for the victims of Ponzi schemer Bernard Madoff can’t pursue big banks for allegedly aiding the fraud, a federal appeals court has ruled.

Trustee Irving Picard had sought to recover “mega-billions” from the banks, the Am Law Litigation Daily reports, arguing that they had ignored red flags of Madoff’s fraud because they wanted to earn banking fees. But the New York-based 2nd U.S. Circuit Court of Appeals ruled on Thursday that Picard can’t sue because of the doctrine of in pari delicto, report the Litigation Daily, Reuters, the Associated Press and Bloomberg.

As trustee, Picard stood in the shoes of Madoff’s firm and he can’t recover money for a fraud orchestrated by that firm, the appeals court said in its opinion (PDF).

The Litigation Daily says Picard was seeking to recover nearly $80 billion from the banks, while Reuters and Bloomberg put the figure at close to $30 billion. The ruling upheld rulings by two lower court judges.