There is a widening circle of prominent Republican donors and activists stepping forward this week to declare that they were audited by the IRS, and many now are questioning if they were targeted for their political views.

"It makes you wonder," said Charlie Moncrief, a Texas oil executive who is raised more than $1 million for Mitt Romney's 2012 presidential bid. "You just don't know. But given what's out there now, you have to ask the question."

The wave of mistrust on the part of prominent conservatives comes in response to a report by the IRS Inspector General's office published Tuesday that suggested the IRS singled out conservative advocacy groups -- specifically those with references to the Tea Party in their names -- for special scrutiny after they had applied for nonprofit status. The report has triggered a federal investigation into whether officials inside the taxing agency let political motives guide their actions.

Now Frank VanderSloot, an Idaho businessman who donated more than $1 million to groups supporting Romney, told ABC News he believes he may have been targeted for an audit after his opposition to the Obama administration. So did Hal Scherz, a physician who started the group Docs4PatientCare to lobby against President Obama's health care initiative, and became a vocal critic of the president on cable news programs. Franklin Graham, the son of the evangelist Billy Graham, said he believes his father was a target of unusual IRS scrutiny as well, according to published reports Wednesday.

Graham told Politico that groups founded by his famous father, the Billy Graham Evangelistic Association and the family's international humanitarian organization Samaritan's Purse, were both subjected to aggressive action by the IRS. In a letter to President Obama, which he shared with the news outlet, he wrote: "I do not believe that the IRS audit of our two organizations last year is a coincidence -- or justifiable."

So far, the suggestions of impropriety at the Internal Revenue Service have been limited to the agency's review of advocacy groups that had applied for nonprofit status. The IRS Inspector General's office has not signaled that political factors could have bled into audits of individuals and the IRS maintained in a letter to the Inspector General that its agents made no decisions "out of partisan or political viewpoint."

The IRS reiterated today that politics "play[s] no factor in audit selection."

"The IRS stresses that audits are based on the information contained on the tax return and the underlying tax law," the agency said. "The audit process is handled by career, non-partisan civil servants, and we have processes in place to safeguard the exam process."

But many of those in an already skeptical group of prominent conservatives are unconvinced.

"I happen to believe there are people inside the IRS who feel emboldened," said Cleta Mitchell, a Washington attorney who represents several of the conservative groups that were audited. "I've heard of several instances of donors to conservative causes who were audited. We need to find out if this is just random or if it's more than that."

Mitchell said she is hearing from a range of high-profile Republicans who want to know if their politics motivated the government's decision to audit them.

"I suspect that they looked at individuals as well," Scherz told ABC News. "It is odd that nothing changed on my tax return and I was never audited until I publicly criticized Obamacare."

VanderSloot said he did not want to jump to conclusions when he first received notice from the IRS last June that he would be audited for the first time in roughly 30 years. The head of a nutritional supplement company, VanderSloot had been singled out during the 2012 presidential campaign for being, in the words of an Obama campaign website, "a bitter foe of the gay rights movement," a claim he says is untrue. VanderSloot has said he raised between $2 million and $5 million for Romney's campaign.

He said when he first heard about the audit, he did not believe he was being targeted because of his political advocacy. "I'm thinking this is America and they're not going to that," he said. Then he learned his wife and business would also face audits, and he grew more concerned.

Now, though, he thinks the matter warrants investigation.

"We can be suspicious, and I am suspicious as I can be about this," VanderSloot said. "But I don't think we ought to let our minds run too far in the field of accusations until we know what the facts are."

Democrats say the mistrust is inevitable given the allegations now facing an already unpopular government agency.

"The IRS has done something no one else has been able to do, and that is create bipartisan outrage," said Mo Elleithee, a partner at Hilltop Public Solutions, a left-leaning consulting firm. "But taking a step back, what may be one of the sad by-products of this whole disaster is that it does give the aura of credibility to those who are looking for reasons to distrust government."