On Sept. 13, 2012, a Libyan man investigates the inside of the U.S. Consulate, after an attack that killed four Americans, including Ambassador Chris Stevens on the night of Tuesday, Sept. 11, 2012, in Benghazi, Libya.

A visitor rings the doorbell to a large gated villa in Benghazi, and a gardener slowly opens the heavy metal door. He welcomes guests with a big smile, offering them tea before giving them a guided tour of the sprawling grounds with its swimming pool and hefty trees, which obscure the view from prying eyes. But the villa is not just another secluded house owned by a wealthy Libyan seeking privacy. It is the most sensitive crime scene in the world.

For each day of the past two weeks, TIME has visited the American consulate in Benghazi where the ambassador and three others were killed on Sept. 11th. And with the passing of every day, people cart off more and more evidence and sensitive information that couldendanger the lives of Americans still in Libya, and impair the FBI investigation into the attack just now getting underway.

After 23 days, Libyan authorities finally cordoned off the consulate Thursday and prohibited people from entering it as American investigators made their first brief visit to the site. For weeks, Washington has said it was too dangerous to send FBI agents to Benghazi. But now, they may be too late to piece together the clues to the attack. A number of Libyans and foreigners have visited the compound, according to the gardener Idris Muhammad Juma’a. “Every day people come and look around,” Idris told TIME, sitting in a plastic chair under a palm tree on Tuesday. “The other day some Turks came and took a big painting.”

But it is not the disappearance of paintings that should disturb American investigators and intelligence officials. Large white boards listing names and numbers of U.S. military and diplomatic installations abroad have vanished. Documents with detailed accounts of previous attacks against the consulate have disappeared as well. “It’s not our job to stop people from taking things,” Idris says. “We are just gardeners.”

Though Idris and his companion are tending to the grounds, the Americans have not secured the compound. Idris saysAmerican officials only visited the consulate once around Sept. 24. “They came with some translators and took some pictures. They left after 10 minutes.”

It is not only the consulate the FBI has neglected. Libyans guarding the compound the night of the attacknote they have not been contacted by Americans since the incident. “I thought they would want to speak to us,” says Muhammad, who saw some of the attackers. “But no one has called us.”

In such investigations, it is routine for FBI agents to wait until they can interview people directly, according to Ali Soufan, the lead FBI agent in the inquiry into the 2000 USS Cole attack. “You don’t collect intelligence over the phone. It must be face to face, not over Skype.” Soufan, who detailed his role in the Cole investigation and his part in uncovering the men responsible for the Sept. 11 attacks in his book The Black Banners, notes that there are several things that have delayed the FBI’s arrival in Benghazi. “You need a country clearance. If you don’t have a [local] partner to work with, how are you going to interrogate?”

But with a Libyan central government too weak to stabilize the country, it has to rely on local militias to protect the consulate and provide security for the FBI. Since each brigade is led by a different man, the Libyan government has had to coordinate FBI visits with a number of individuals. And because the FBI needs a “significant team that includes different specialties ranging from forensics teams to communication specialists” according to Soufan, the group needs a level of security no one in Benghazi can provide.

“It’s just not safe for the Americans to come now,” says a leader of one of the brigades in touch with the American embassy in Tripoli.

In the interim, evidence at the consulate has been tampered with. It is clear that rubble in the annex where Ambassador Chris Stevens took refuge has been moved. “Securing the crime scene is very important to prevent contamination,” says Soufan. But with so many people rummaging through the consulate in the past three weeks, ranging from curious visitors to possibly the attackers themselves, the compound is being compromised daily. And that will make the FBI’s job all the more difficult when it finally pursues its investigation in earnest.