WASHINGTON -- Herman Cain's rise in the Republican primary polls several months ago was always viewed warily by the GOP establishment. Many believed the longtime businessman was more engaged in his book tour than his presidential bid. And so while the crowds on the campaign trail may have liked what they heard, at some point, the thinking went, the fad would fade.

At this point, it can't fade quickly enough. On Monday, Ginger White, a longtime friend of Cain's, told an Atlanta-based Fox affiliate that they had engaged in a consensual, 13-year-long extramarital affair. Jumping in front of the story, Cain denied the allegation in an interview with CNN before the Fox report aired. But as White becomes the third woman to level accusations of sexually inappropriate behavior at Cain, some inside the GOP tent -- worried about the side effects the "circus" is having on the party at large -- are no longer willing to give him the benefit of the doubt.

"Cain is irrelevant, and the quicker he gets out of the race the better it will be," said Ed Rollins, a longtime Republican strategist. "My fear is that he marches to the beat of his own drum and he may try to drag it on and deny and deny and deny. And my sense is that will likely be the pattern here. But there is no way he can be the nominee of our party. The quicker he gets out, the better for him and for us."

"As people freeze their impressions going into the holidays and early voting events, the impressions right now of all politicians, and not just ours, are not good," he added.

Rollins is not an impartial voice. In the earlier stages of this presidential primary campaign, he served as Rep. Michele Bachmann's (R-Minn.) campaign manager and chief political adviser.

Since leaving that post late this summer, he has been a ubiquitous presence on cable news, generally offering a fairly sober analysis of the primary and even subtly suggesting that his former client is, herself, a flash in the Republican primary pan. Rollins is not the only top Republican suggesting that Cain's candidacy could cause harm to the GOP's brand.

When the first round of harassment allegations occurred, Republican lawmakers and strategists alike said they would withhold criticism of Cain until more information was revealed. But even then, some expressed unease -- and in some cases, disdain -- toward the image he was projecting onto the rest of the field.

“I am deeply skeptical about Cain’s candidacy on the substance and policy level,” Fergus Cullen, a former chairman of the New Hampshire Republican Party, told The New York Times. “He’s gotten a free pass from his peers and the press until very recently – call it the soft bigotry of low expectations – because no one took him seriously or considered him a threat.”

Now, with a third woman going public, even Cain's defenders have grown tired.

"As someone who has been for Cain since he was an asterik this is all getting tough to handle," tweeted Chris Barron, a GOP strategist and board chair of the Republican gay-rights group GOProud.

The idea that the time has come for Cain to bow out of the GOP race is not one the candidate himself seems willing to accept. He told CNN Monday he was "not dropping out of this race" so "long as my wife is behind me, and as long as my wife believes that I should stay in this race."

Certainly, there is enough buzz surrounding his candidacy to keep him going. While the former Godfather's Pizza CEO has fallen in the polls since the first round of allegations surfaced, he remains the preference of a decent chunk of primary voters.

Whether those numbers deteriorate further seems tied, in part, to the impact the latest allegations will have. But even if they prove damaging, not everyone thinks the GOP suffers from having Cain around.

"I don't think it taints the party at large," said Larry Farnsworth, a GOP consultant and former aide to former House Speaker Dennis Hastert (R-Ill.). "I do think it distracts from having an actual conversation on issues."

Others were more eager to rally to Cain's defense. Dick Morris, the longtime political operative who has been paid by the Cain campaign in the past, told a skeptical Sean Hannity on Fox News that the whole thing was "an old-time vanilla scandal" while accusing White of being a "serial litigator." (In 2001, White filed a sexual harassment claim against an employer.)

Morris is certainly aware of the political perils involved in allegations of sexual misconduct. So too is the man he often advised, Bill Clinton, who managed to navigate a presidential primary while dealing with questions and accusations on par with those Cain faces. But it's not the 42nd president who may inspire Cain to stay in the race; rather, it's the Republican candidate who has benefited most from his downfall.

"Look, Herman Cain is a joke," said one top Republican operative. "Obviously, where there is smoke there is fire, and he needs to make a decision about whether or not he stays in the race. But I do find this extremely interesting, that Cain might get pushed from the race but Newt Gingrich is one of the top frontrunners."

"I don't know how his past [with multiple divorces and extramarital affairs] is not a distraction," the operative continued. "I guess Gingrich's story is pretty much of the public record. Everyone knows it. But since [Cain's problem] is obviously something that happening currently, that makes it not presidential?"