America’s first Black Attorney General Eric Holder is leaving his post after six controversial years hopefully to be replaced by America’s second Black Attorney General Loretta Lynch. As head of the Justice Department, Holder was America’s chief law enforcement officer during that time, making sure in the wake of situations like Ferguson that the law of the land was upheld on the federal level.

For his work to make the justice system more equitable, Holder was honored with an NAACP Image Award this past week. He says it was an honor he won’t soon forget.

“That was a real honor for me to be recognized by the NAACP. They’ve done a lot for the country over the course of century and I was really heartened that Chairman Brock decided to recognize me in the way that she did. It was a lot of fun and it was a great honor.”

Despite his ascension to the top job in law enforcement, Holder says becoming the nation’s first Black Attorney General was not something he consciously planned.

“I went to law school undecided about what I wanted to do and then I kind of fell in love with criminal law, but I never envisioned being the Attorney General,” says Holder. “I had an interest in criminal law and I became a prosecutor and then joined the Justice Department but if somebody told me back in 1976 when I first got here that I would lead the Department, I would have been a little incredulous.”

Holder says that he was ready by the time he was asked by then President-elect Obama to take on the role. Although he’s been heavily criticized by Republicans for advancing an agenda much more progressive than they wanted, he’s also been criticized by progressives who thought he should have done more to make The Justice Department more fair, particularly as it relates to African-American men.

“I’m real proud of the men and women of the department. In terms of our criminal justice reform efforts, I think that combating hate crimes, what we’ve done with LGBT citizens and equality for them, defending the right to vote, working with national security issues in a way that’s consistent with our values, not torturing people and restoring the morale of this department. People tend to orget the state of the Justice department when I got here. I think I’m leaving this department in the way I want to with morale high and real good sense of mission.”

In six years, Holder achieved much of the agenda he set out to, but one glaring disappointment remains a part of his record.

“I think our inability to enact meaningful gun safety measures in the wake of the Sandy Hook massacre, especially because the American people wanted it to happen,” Holder says. “But Congress couldn’t get beyond the pressure of the gun lobby. I really regret that we didn’t make something better out of that tragedy.”

Holder can’t leave his post, of course, until nominee Lynch is confirmed, something that is ongoing. Lynch is a longtime family friend, as she and Holder’s wife, Dr. Sharon Malone, were among the first women to pledge Delta Sigma Theta at Harvard University.

Was Holder surprised to hear his name so much during the Lynch’s confirmation hearings?

“I was wondering if this was Loretta’s confirmation hearing or a vote on Eric’s stewardship of The Justice Department,” Holder says. “Especially day two, when Senator Leahy asked if there is anyone here who doesn’t think Lynch should be confirmed, raise your hand and nobody raised their hand and I thought ‘Why are they here then?'”

Holder, now 64 says he’s not exactly ‘retiring.” Married to Malone since 1990, the father of three says his wife refers to his next moves as ‘transitioning.’

“I have kids in college and one going to college so I want to stay involved in the work that I’m doing. I’m going to take some time off to recharge but I want to come up with a way to stay involved in civil rights in particular reconciling law enforcement with the communities that we serve and with a continued emphasis on voting rights. These are all the things I want to continue to do.”