Federal authorities are investigating a potential case of human trafficking at the suburban Washington home of a Saudi diplomat, officials said Wednesday.

Authorities were called Tuesday afternoon to an upscale neighborhood in McLean, Virginia, where they "removed two potential trafficking victims," U.S. Immigration and Customs Enforcement spokesman Brandon Montgomery told CNN.

A State Department official said the McLean house is the home of the Saudi military attaché.

The case is under investigation by Homeland Security Investigations, a unit within ICE.

The two women, who are from the Philippines and currently work at the Saudi Embassy in Washington, claim they were mistreated, according to the State Department official.

JUST WATCHED

Two women removed from diplomat's home

MUST WATCH

Two women removed from diplomat's home03:02

The women are charging, according to a different State Department official, that the Saudi attaché kept their passports, made them work extremely long hours and did not pay them. The source said they had not seen anything to indicate the women were physically harmed.

The official also pointed out these allegations are similar to several other cases they heard from domestic workers who work for diplomats from the Persian Gulf.

According to ICE, cases like this are very "victim-centric," meaning that once there is "confirmation of suspicions," authorities immediately go in and remove the victim and then begin the investigation.

There was no word on the identity of the Saudi diplomat who lived there.

The Saudi Embassy in Washington did not respond to requests for comment.