The mob of anti-American protesters marched forward under a blazing sun — their faces scowling, their fists raised in the air. "Down with America," they chanted.

The nearly two dozen demonstrators entered the walls of a town surrounded by desert. They kept chanting as they walked past a series of low-slung sand-colored buildings and gathered in front of a group of U.S. soldiers.

"Go home, Yankees!" the protesters hollered.

The scene looked straight out of a restive American-occupied city in the Middle East. But this was nowhere near Mosul or Kabul, and the protesters were not harboring real-world gripes.

The setting was the U.S. Army Fort Irwin National Training Center in eastern California. The mob was made up of mostly immigrant role players, organized by the Pentagon and paid for by American taxpayers.

Over the past 12 years, the U.S. government has hired role players and sent them to bases like Fort Irwin to help U.S. soldiers prepare for real-life situations overseas.

The program has fueled a massive cottage industry of companies paid handsomely to funnel a steady stream of predominantly Afghan and Iraqi immigrants, many of whom are now U.S. citizens, to American military trainers.

But a year-long NBC News investigation found that the the government's own watchdog office has never conducted a careful examination of the more than $250 million-a-year program.

A review of hundreds of pages of government documents, as well as interviews with a half-dozen military and training officials, revealed that the Government Accountability Office has not undertaken a comprehensive audit of the more than 250 companies supplying the role players at an annual cost of roughly a quarter billion dollars.

Independent watchdog groups said they were troubled by the under-the-radar nature of the program.

"We've seen this a lot over the last 20 years now where we have these ad hoc programs that sprang up to meet an immediate need but they don't receive quite enough scrutiny," said Dan Grazier, a defense researcher at the Project on Government Oversight's Center for Defense Information.

Aaron Scherb, the legislative affairs director at the nonprofit government watchdog Common Cause, said a comprehensive review of the program is long overdue especially in light of the Trump administration's decision to pull American troops out of countries like Afghanistan and Syria.

"I think a GAO audit on the effectiveness of the program as well as a potential Department of Defense inspector general report would go a long way to determine how necessary and useful this program is, as the U.S. military continues to draw down in certain countries and regions around the world," Scherb said.

A spokesman for the Government Accountability Office confirmed that it has not examined the military's practice of using civilians to simulate the role of villagers in training exercises. A spokesperson for the Department of Defense inspector general said there was no record of the office ever having reviewed the program.

The military isn't the only arm of the government that hires role players. Tens of thousands are also used in training exercises conducted by the FBI and CIA, as well as local police departments, according to a review of government contracts.

Hiring the role players to act as civilians in conflict zones offers military trainers the opportunity to test soldiers in fluid situations that mirror real-world encounters, according to Col. Robert E. Lee Magee.

"If we don't give them an opportunity to be exposed to that here and now, they'll struggle with it for real," Magee said during a training exercise last summer.

The role players act as human props in situations that can range from the aftermath of a terror attack to the questioning of a suspect in detention.

Magee noted that the majority of the role players hail from countries that the soldiers are likely to deploy to. "In every possible way, the National Training Center attempts to do its utmost to replicate the conditions that the U.S. Army faces in the world today," he said.

For those selected to work at Fort Irwin, the appeal is obvious.

They work two-week rotations 10 times a year. The pay: up to $50,000 for 20 weeks of employment.

Over the summer, some 120 role players were living in the mock village located on the base's grounds in the middle of the Mojave Desert.

Many of them were reluctant to give on-camera interviews or even reveal their names. They feared possible retaliation against their family members back home.

But one of the actors, Salim Al-Jumaaily, ultimately agreed to give NBC a tour of his home inside the mock village known as Eugen, which has electricity but no running water. The apartment is large by New York City standards, with a refrigerator, an AC unit, and satellite TV with numerous Arabic channels.

"I have everything almost," Al-Jumaaily said, "except my wife."

Another actor, who declined to give his name, said the work gave him more than a steady income. He sees the job as performing a vital service to the country.

"This is about saving lives," the man said. "I'm doing this for the U.S. Army."

Army officials told NBC News the training was particularly helpful after the 9/11 attacks when troops were deployed to countries like Iraq and Afghanistan and forced to patrol towns and cities inhabited by people with unfamiliar cultures.

Grazier, the defense analyst, served in the Marines for 10 years and trained with role players ahead of planned deployments to Iraq and Afghanistan. He said the experience was invaluable.

"You can learn a lot from books but until you sit down with people from that culture, only then do you gain the knowledge and experience necessary to be really successful in a counterinsurgency campaign," Grazier said.

Fort Irwin is home to a dozen mock villages used for training exercises. But it's not the only base that offers soldiers the chance to interact with role players.

In all, there are more than 200 mock villages in at least 100 bases across the country, including Camp Pendleton in California, Camp Upshur in Virginia and Fort Stewart in Georgia.

The largest village at Fort Irwin, known as Razish, is as elaborate and carefully designed as a Hollywood set. It features more than 500 buildings, signs with Arabic writing hanging on several walls and a central market complete with barrels of fake fruit.From a hilltop control center, training officials can pump smoke and audio into the dwellings to simulate the real thing.

The exercises with role players force the service members to improvise.

"We script major events, but the individual actions between the soldier and the civilians, that's not scripted," said Lt. Col. Gerald Rebeshini. "We can't get that specific."

The role player training doesn't always bear fruit for the soldiers. The unit that trained at Fort Irwin over the summer was not shipped off to the Middle East. The soldiers are currently stationed in South Korea.