Attorney General Eric Holder is challenging Republicans who are calling for his and President Obama's impeachment, and denouncing what he calls a "gridlocked Washington" stalled by what he says is a Republican Party bent on blocking any of the administration's efforts.

“For whatever reason, [some] Republicans decided early on that this was a president they were just simply not going to cooperate with,” Holder said in a rare interview with ABC News' Pierre Thomas. "And over the past five-and-a-half years, we have seen demonstrations of that, where the president has reached out his hand, offered compromises that have simply not been met [in the way] they have been in the past by a Republican Party willing to do the appropriate things."

Administration efforts to pass comprehensive immigration reform, for example, have failed. Asked about calls by Sarah Palin to impeach Obama over the administration’s immigration policies, Holder said: “She wasn't a particularly good vice presidential candidate. She's an even worse judge of who ought to be impeached and why.”

Holder similarly dismissed calls for himself to be impeached for declining to appoint a special prosecutor to investigate the IRS scandal. Holder insisted that a special prosecutor isn’t necessary, with “career people” and FBI agents “doing a good, professional job” investigating the matter.

As for House Speaker John Boehner’s lawsuit against the administration over the Affordable Care Act, Holder said he doesn’t think “that lawsuit's going to have legs.”

“It's a more, I think, a political gesture than a truly legal one,” he said. “Filing a lawsuit against the president that has no basis is not going to improve the quality of life for the American people.”

In the wide-ranging interview for “This Week with George Stephanopoulos,” Holder spoke at length about race and discrimination in America, saying the country is in “a fundamentally better place than we were 50 years ago.”

“We've made lots of progress,” he said. “I sit here as the first African-American attorney general, serving the first African-American President of the United States. And that has to show that we have made a great deal of progress.

“But there's still more we have to travel along this road so we get to the place that is consistent with our founding ideals,” he said.

Holder said that he and President Obama are treated differently than their predecessors.

“There's a certain level of vehemence, it seems to me, that's directed at me [and] directed at the president,” Holder said. “You know, people talking about taking their country back. … There's a certain racial component to this for some people. I don’t think this is the thing that is a main driver, but for some there's a racial animus.”

Asked about his controversial comments from 2009, in which he called the United States a “nation of cowards” when it comes to race, Holder stood firm.

“I wouldn't walk away from that speech,” Holder said. “I think we are still a nation that is too afraid to confront racial issues,” rarely engaging “one another across the color line [to] talk about racial issues.”

In addition, Holder took Republicans to task for efforts to, among other things, enact voter ID laws in some states. He called such moves “political efforts” aimed at making it “more difficult” for “groups that are not supportive of those in power” to “have access to the ballot.”

“Who is disproportionately impacted by them? Young people, African Americans, Hispanics, older people, people who, for whatever reason, aren't necessarily supportive of the Republican Party,” Holder said, noting that “this notion that there is widespread in-person voter fraud is simply belied by the facts.”

Holder said the Justice Department is expected to soon file challenges to restrictive voting laws in Ohio and Wisconsin, as the department already did in Texas and North Carolina.

“I'm attorney general of the United States. I will not stand for -- I will not allow people to take away that which people gave their lives to give, and that is the ability for the American people to vote,” Holder said.

In the interview, Holder was also asked about the controversy now surrounding the name of Washington's professional football team, the Redskins.

“I think the name ought to be changed,” Holder, a long-time Washingtonian, said. “I think it is an offensive name. And the Redskins are … a team with a storied history that has huge amounts of support in Washington, D.C. And I think in the 21st Century, they could increase their fan base, increase their level of support if they did something that is so, from my perspective, so obviously right.”

Like "This Week" on Facebook here. You can also follow the show on Twitter here.