Privacy advocates said the government enlisting Yahoo to assist in email monitoring would be wrong.

"The order issued to Yahoo appears to be unprecedented and unconstitutional," said an attorney for the American Civil Liberties Union in a statement. "It is deeply disappointing that Yahoo declined to challenge this sweeping surveillance order."

Other legal experts aren't so sure. The whole issue underscores a legal gray area that the tech industry and the U.S. government have yet to clarify, said Roy Hadley, a lawyer at Thompson Hine who studies privacy issues.

"Whether it's unconstitutional is one of those things that's open to debate," he said. "We really don't have any national law that has been interpreted on that subject."

It's no secret that the U.S. routinely requests user data from the tech industry to help with its surveillance efforts. Former NSA contractor Edward Snowden exposed more details of the government's practices in 2013.

The Yahoo initiative may have gone beyond other information requests by being more broad in scope. Reportedly, the company was searching through the incoming emails of all its customers in real time, meaning it may even have been spying on users outside the U.S.

It's possible this violated privacy laws in other countries, Hadley said. "Europe has far more robust privacy rights than the United States," he said.

Cybersecurity experts also wonder whether Yahoo complied with similar requests from foreign governments.

"We don't know if Yahoo is allowing other governments to also do this surveillance," said Jeremiah Grossman, chief of security strategy at SentinelOne. "It sounds to me like they would."

Yahoo hasn't publicly provided any details of the program or even confirmed its existence. On Tuesday, the company simply said, "Yahoo is a law-abiding company and complies with the laws of the United States."

Yahoo's mass email searching program could be one of several already in place, said Michael Sutton, CISO at security firm Zscaler. Internet companies increasingly are encrypting their data, making it harder for law enforcement to extract information through its own surveillance programs.

"They can't defeat the encryption, so they really have no choice but to make the service provider their partner in this," he said. "I think you are going see more programs like this because the intelligence community can't do this on its own."