As top House Republican leaders remain silent on the unfolding scandal surrounding Rep. Scott DesJarlais of Tennessee, conservatives in Congress are starting to groan that his infidelities add up to bad optics for the group.

Still, they are stopping short of calling for his resignation or asking that he step down from the Republican Study Committee, which identifies itself as the conservative conscience of the House GOP.

Rep. Dennis A. Ross of Florida said he considers DesJarlais a good friend, but said the revelation that DesJarlais’ then-wife had two abortions and that he had several affairs, including with patients he saw at his medical practice, makes other anti-abortion lawmakers look bad.

“I think it has an adverse impact, yeah,” Ross said. “I think it has to play itself out, I really do. I don’t know the facts of it, I just know what I’ve read and it’s disappointing. And quite frankly, I firmly believe it’s between him, his wife and his God.”

Rep. John Fleming, co-chairman of the anti-abortion Congressional Values Action Team, said the group does not go after members for their own actions, but works instead to influence policy.

“If there’s been misconduct in any way or problems, that’s ultimately a decision that Dr. DesJarlais’ constituents are going to have to deal with and he’s going to have to be accountable for,” Fleming of Louisiana said.

Republican members of the Tennessee delegation have largely remained silent. When asked, Reps. Marsha Blackburn, Chuck Fleischmann and Phil Roe all said they did not want to comment because they had not read the Chattanooga Times Free Press report.

The report, culled from sworn testimony in court documents during DesJarlais’ 2001 divorce proceedings, showed that he had “sexual relationships with at least two patients, three co-workers and a drug representative while he was chief of staff at Grandview Medical Center in Jasper, Tenn.,” and that he “supported his ex-wife’s decision to get two abortions before their marriage.”

“I don’t know that it reflects badly,” Roe said, when told about the report. “I think it’s an individual decision that someone’s made.”

“I concentrate on my work in the 3rd District. Dr. DesJarlais and the voters of the 4th District have spoken, he won,” Fleischmann. “I’ve not heard anything adverse from anybody.”

A top conservative aide, however, said members may be more fed up than they are letting on.

“Many members are not at all happy that their efforts for traditional values get weakened by their own colleagues’ personal actions to the contrary,” said the aide.

Still, Rep. Steve Scalise of Louisiana, who was elected Thursday to serve as the chairman of the RSC during the 113th Congress, said DesJarlais is welcome in the group as long as he pays his dues.

“It’s my understanding he’s already rejoined the RSC,” Scalise said. “I mean, this is an organization of conservatives that unite to pass conservatives solutions to our country’s problems. We’re going to all be focused and united on getting things done.”

Rep. Jack Kingston, who represents neighboring Georgia, said Congress has a full workload and an issue like this should really be hashed out in the district, rather than in the Capitol.

“I think that with these types of dustups, you sort of have to let it work through its own system and right now, with him just getting re-elected with pretty good numbers, and there being so many other things to worry about, I don’t think there’s any temperature to for anybody to get involved in a Tennessee issue for the time being,” he said. “People have to reach their own conclusions about abortion. ... If the pro-life movement was dependent on people who have always been pro-life it would be a small movement. But there’s plenty of former pro-choice people in the movement.”