The FBI agent who spurred the investigation that turned up David Petraeus' affair was taken off the case because authorities grew concerned about his relationship with one of the key figures in the scandal, Fox News has confirmed.

The new wrinkle, first reported by the Wall Street Journal, seems to fill in some of the gaps in how the investigation developed and how word of the investigation made its way to lawmakers, the White House and eventually the public last week, when Petraeus resigned as CIA director.

The FBI agent under scrutiny had launched the investigation into harassing emails sent to Jill Kelley, a Petraeus family friend, but the agent was removed from the case over the summer because of his behavior, which included sending shirtless photos of himself to Kelley, the Wall Street Journal reported, citing officials familiar with the probe.

The agent's identify wasn't divulged, but he now faces an internal investigation for his behavior, the Journal reported.

Also late Monday, an FBI spokesperson confirmed to Fox News agents were searching Broadwell's North Carolina home, though they did not say what they were doing there or what prompted the search.

The Associated Press reports FBI agents appeared at Broadwell's home carrying the kinds of cardboard boxes often used for evidence gathering during a search. They walked through the open garage of Broadwell's house and knocked at a side door before entering the home, but refused to talk to the media.

Fox News confirmed Sunday that the investigation started when Kelley, 37, alerted the FBI about the harassing emails, which appeared to be an attempt to blackmail Petraeus, sources said. But there initially were questions over where those emails came from, with early indications that they might not have come from Petraeus biographer Paula Broadwell, with whom he was carrying on an affair.

However, Fox News can now confirm from multiple law enforcement sources that those emails came from multiple dummy accounts, which were traced back to Broadwell. The reason the FBI had jurisdiction is because cyber-harassment is a federal crime. And once the FBI got to Broadwell, they uncovered the affair.

Kelley, had complained about the emails in May to her friend, the FBI agent, who referred it to a cyber crimes unit. The Wall Street Journal's sources said the agent appeared to become obsessed with the matter and then was prohibited from participating in the investigation.

He later contacted a member of Congress, fearing that FBI leaders would sweep the matter under the rug, the Journal reported.

That wold seem to coincide with reports that the first knowledge of the affair outside the FBI came from an agency whistle-blower who contacted a Capitol Hill Republican, who then told House Majority Leader Eric Cantor. Rep. Dave Reichert, R-Wash., got a tip from a friend who knew the whistle-blower, but it wasn't immediately clear if the FBI agent was the whistle-blower.

Cantor staffers said they didn't immediately tell the House Intelligence Committee or chamber leaders because they didn't know whether the tip was credible.

No security breaches appear to have occurred in the email exchange, but Capitol Hill lawmakers expressed shock and disappointment about the stunning revelations and said Petraeus made the correct choice in resigning.

Kelley, a Tampa, Fla., resident who is married with three children, and sister Natalie are close friends of the Petraeus' and spent holidays together. And she was not having an affair with the retired, four-star Army general, sources close to the family told Fox News.

Kelley issued a statement through Smith and Company, a Washington communications and crisis management firm, asking for privacy. She said she respects Petraeus' privacy and wants the same for her own family.