WASHINGTON (CNN) -- Race and politics are a combustible combo that explodes into headlines when an ex-president lights the fuse, as Jimmy Carter did recently.

President Obama during the 2008 campaign faced questions over race and politics.

"When a radical fringe element of demonstrators and others begin to attack the president of the United States as an animal or as a reincarnation of Adolf Hitler or when they wave signs in the air that said we should have buried Obama with Kennedy, those kinds of things are beyond the bounds," the Democrat told students at Emory University on Wednesday.

"I think people who are guilty of that kind of personal attack against Obama have been influenced to a major degree by a belief that he should not be president because he happens to be African-American," he added.

The controversy erupted this week when Carter first raised the race issue to NBC.

"An overwhelming portion of the intensely demonstrated animosity toward President Obama is based on the fact that he is a black man," he said.

Bill Cosby, a black comedian and actor, said Wednesday in a written statement that he agrees with Carter.

"During President Obama's speech on the status of health care reform, some members of Congress engaged in a public display of disrespect," he said.

CNN contributor David Gergen said that some of the allegations of race-baiting might have some weight among Democratic voters.

"Jimmy Carter, I'm afraid, is not alone in his views," he said.

But Gergen warns that the racists tend to be on the fringes of the right and do not reflect on the greater field of opponents to Obama's ideology.

Don't Miss

"But I think it's wrongheaded, and I think it's unfair, and I think it's indeed a libel upon many of the opponents, most of the opponents of health care reform, to say that they're racist," he said. "Seven previous presidents have tried to bring health care reforms of this kind. All seven have failed. And, as I recall, all seven were white."

Carter's comments, though, are the kind that raise people's defenses. In particular, they turn off independents, who by nature tend to hate the hard edges of politics.

The White House, for its part, wants no part of the Carter controversy.

"The president does not believe that that criticism comes based on the color of his skin," said White House press secretary Robert Gibbs.

As a candidate, Obama understood the political danger in letting his race become a major topic. He largely avoided it when he could, but race was always a subtext, as it is now in his presidency.

"But I can also say, frankly, that this White House and even his campaign were very afraid to even go down this road dealing with anything with race," said CNN contributor Roland Martin.

Carter also told NBC that there is an "inherent feeling among many people in this country that an African-American ought not to be president and ought not to be given the same respect as if he were white."

Framing criticism as racism cropped up several times during the campaign.

When Geraldine Ferraro, a Clinton supporter, said during the 2008 Democratic primaries that Obama would never have gotten as far as he had if he had not been black, candidate Obama pointedly left race out of it.

"I think that her comments were ridiculous," Obama said at the time.

But Ferraro blamed Obama supporters for her hate mail, saying "I have been called all kinds of names. And the attacks are ageist. They're sexist. They're racist." iReport.com: Freedom from speech?

And the topic of race even came to haunt Bill Clinton, the so-called first black president.

Before the New Hampshire primary in January 2008, Clinton told an audience that "there's no difference in [Obama's] voting record and Hillary's ever since. Give me a break. This whole thing is the biggest fairy tale I have ever seen."

Some interpreted that as belittling Obama's campaign. After much criticism, Clinton later said he was talking about Obama's opposition to the war in Iraq, not his campaign.

Some observers said Carter's statement was far too broad, and that many of the attacks on Obama are about policy, not race.

"I don't think one can paint a broad brush in saying that all of these folks who have criticism, anybody who criticizes the president ... is based upon the issue of race," Martin said.

Mary Matalin, a CNN contributor and Republican strategist, said Carter's criticism is "absurd."

"It's very dangerous politics," she said. "Barack Obama got, in the last election, more white male voters than any of his predecessors. ... Republicans and conservatives and Democrats and liberals, the whole country felt very good putting -- setting aside policies after his election."