WASHINGTON (Reuters) - Lawyers for the U.S. Justice Department and for congressional Republicans told a federal judge on Tuesday they were in talks to settle a suit stemming from Operation Fast and Furious, a botched probe into gun trafficking to Mexican drug cartels.

While far from certain, a settlement would bring a quiet end to a political furor that stirred the passions of gun owners, ended some Justice Department careers and led Republicans to find U.S. Attorney General Eric Holder in contempt.

The lawyers said they were uncertain whether the talks would be successful, but that they met on Monday and will meet again.

Republicans in the House of Representatives sued Holder, head of the Justice Department, in August to enforce a subpoena to obtain documents pertaining to the probe.

The contempt vote came in June when Republicans accused the Obama administration of withholding documents related to how it responded to the Fast and Furious operation, named after a movie about auto racing.

U.S. District Judge Amy Berman Jackson encouraged settlement talks during a 30-minute hearing on the case in federal court in Washington, D.C.

Federal judges have rarely addressed the complicated questions raised in the case about relations between the White House and Congress, Jackson said. One side or the other might not want to live with a potential "all or nothing" decision that a settlement would avoid.

"There may be some benefit in trying to work it out, either among yourselves or with the help of a mediator," she said.

A lawyer for House Republicans, Kerry Kircher, said he expected the sides to meet again in the near future, but added: "I cannot express any expectation about the outcome of that."

A settlement would be "the appropriate course here," said Justice Department lawyer Ian Gershengorn.

UPROAR OVER OPERATION

Congressional uproar over Fast and Furious led to at least three high-level resignations or retirements from the Justice Department or the U.S. Bureau of Alcohol, Tobacco, Firearms and Explosives.

Fast and Furious was launched in 2009 near the U.S.-Mexico border in Arizona as a response to gun trafficking that fed violence among drug cartels.

Federal agents who ran the operation focused on building cases against the leaders of a trafficking ring, and in the process did not pursue low-level buyers of about 2,000 potentially illegal firearms.

The operation was brought to light when a U.S. Border Patrol agent, Brian Terry, was killed in December 2010 in Arizona. Two guns connected with the case were found at the scene of the shootout where he died, although investigators could not determine if the guns were used to kill Terry.

The operation became a rallying point for gun owners and dealers, who suspected the operation misused Arizona gun shops and was motivated by Holder's support for gun control.

A report in September from the Justice Department's own inspector general found inadequate supervision of the operation and faulted the operation's tactics.

It also cleared Holder of wrongdoing, finding no evidence he knew of the operation before details became public.

Judge Jackson, an appointee of President Barack Obama, disclosed connections to two lawyers involved in the suit and said this should not require her removal from the case, but either side could request it.

She said the two - Kircher, the Republicans' lawyer, and Deputy Attorney General James Cole, author of a letter that is central to the lawsuit outlining Obama's executive privilege claim - were both references for Jackson prior to her nomination for a judgeship, and Cole took part in her swearing-in ceremony.

"This is a major metropolitan area, but sometimes it has the feel of a small town," she said.

(Reporting by David Ingram; Editing by Howard Goller and Todd Eastham)