Holder on Sunday told ABC's "This Week" that "there's a certain level of vehemence, it seems to me, that's directed at me (and) directed at the President. ... There's a certain racial component to this for some people."

Holder said he also stood by comments from a 2009 speech in which he said America is a "nation of cowards" on race and sees a crop of new voter identification laws, many of them passed by GOP-controlled state legislatures, as a way to disenfranchise minority voters.

Some conservative figures accused Holder and the White House of "playing to the fears and anxieties of minorities in hopes of scaring them to the polls in November," as Wall Street Journal opinion writer Jason L. Riley wrote in an editorial for the paper on Monday.

Racism directed at the President

Warren; Impeachment; Paul vs. Perry

Holder, GOP dismiss impeachment

"The willingness of the Obama administration to stoke racial divisions for political gain is astounding," Riley wrote adding that the voter identification laws are intended to ensure "ballot integrity" and that there's no proof they suppress minority votes.

Former Illinois Republican congressman turned conservative radio host Joe Walsh Tweeted: "Simple question for Eric Holder and all others who use race to protect Obama: Is it possible to oppose Obama and not be a racist?"

The Washington Times, a conservative newspaper, said Holder had "thrown down the race card."

Both sides are echoing a refrain sounded throughout Obama's presidency but the volume is so loud that neither side can hear what the other is saying, experts say.

For example, Holder's claims speak to subtle institutional racism that is sometimes harder for those in the majority to recognize.

"That's how racial animus or hostility works -- it doesn't require people to be conscious about it," said Imani Perry, an African-American studies professor at Princeton University.

As attorney general, Holder is likely speaking to some sentiment held by the Obama administration, said Mark Anthony Neal, an African-American studies professor at Duke University.

"I rarely think of Eric Holder being rogue in his comments," Neal said. "He is just the one who is allowed to speak that way publicly."

And not all conservatives are motivated by race, said Andra Gillespie, an associate professor of political science at Emory University.

"I have no doubt there are conservatives who disagree with the president on policy grounds," Gillespie said. "And I think people need to examine their hearts sometimes and examine the ways in which stereotyping informs the ways we view things."

All of this means that voters should take a long and introspective look at what motivates their reactions to politicians and policies, experts say.

"Bias can manifest itself in lots of ways. And nobody gets a free pass on this," Gillespie said. "Sometimes you are more likely to be supportive of people who are like you and less likely to help people who are not like you. When that manifests in the workplace or in politics it is an issue."