“There's a certain level of vehemence, it seems to me, that's directed at me [and] directed at the president,” Holder told ABC. “You know, people talking about taking their country back. … There's a certain racial component to this for some people. I don’t think this is the thing that is a main driver, but for some there's a racial animus."

Holder said the nation is in “a fundamentally better place than we were 50 years ago.”

“We've made lots of progress,” he said. “I sit here as the first African-American attorney general, serving the first African-American president of the United States. And that has to show that we have made a great deal of progress.

“But there's still more we have to travel along this road so we get to the place that is consistent with our founding ideals,” he said.

He also stood by his controversial comments made during Obama’s first year in office in which he said the U.S. was a “nation of cowards” when it comes to race.

“I wouldn't walk away from that speech,” Holder said. “I think we are still a nation that is too afraid to confront racial issues,” rarely engaging “one another across the color line [to] talk about racial issues.”

The attorney general also pointed to Republican efforts to enact stricter voter ID laws in southern States as evidence that more needed to be done to protect minority rights. Republicans have maintained the efforts are designed to prevent voter fraud, while Democrats say instances of fraud are exceedingly rare, and far outpaced by the minority population that does not have identification that would be unable to vote.

Holder called the laws “political efforts” designed to make it “more difficult” for “groups that are not supportive of those in power” to “have access to the ballot.”

“Who is disproportionately impacted by them? Young people, African-Americans, Hispanics, older people, people who, for whatever reason, aren't necessarily supportive of the Republican Party,” Holder said, adding that “this notion that there is widespread in-person voter fraud is simply belied by the facts.”

Holder said the Justice Department was planning legal challenges of new voting laws in Ohio and Wisconsin. It has previously filed suit in Texas and North Carolina.

“I'm attorney general of the United States. ... I will not allow people to take away that which people gave their lives to give, and that is the ability for the American people to vote,” Holder said.