The two banks, which could not immediately be reached for comment, don't have U.S. operations. So investigators got authorization to serve the summonses on four U.S. banks and one London-based bank where Zurcher Kantonalbank and N.T. Butterfield maintained correspondent accounts to service U.S. clients.

Martin Schulz, president of the European parliament, told CNBC that growing concerns over tax evasion was leading to a loss of trust among the European public.

The court approvals authorize so-called John Doe summonses that the IRS has used to obtain information about possible tax fraud by individuals whose identities are unknown. The tax agency and Department of Justice has relied on the legal tactic during their continuing crackdown on offshore tax evasion.

"These John Doe summonses will provide information about individuals using financial institutions from Switzerland to the Cayman Islands to Hong Kong to avoid their U.S. tax obligations," said Assistant U.S. Attorney General Kathryn Keneally.

Authorities previously used similar summonses in 2011 to seek information about American clients of London-based HSBC's India division.

They also won court approval in 2009 to serve John Doe summonses on Swiss banking giant UBS. That ultimately led to UBS turning over information on an estimated 4,450 American clients. The bank, Switzerland's largest, also paid a $780 million fine under a deferred prosecution agreement after acknowledging it had held clients duck U.S. taxes.