House Majority Whip Jim Clyburn (D-S.C.) has apologized after accusing his fellow Democratic leaders of engaging in “tokenism” and failing to prioritize diversity in its staffing practices.

Clyburn took aim at House Speaker Nancy Pelosi and House Majority Leader Steny Hoyer over what he called a lack of diversity among top staff in their offices. The 14-term South Carolina congressman quickly walked back his remarks, however, after commenting to The Wall Street Journalthat “tokenism is all right with them.”

Clyburn later clarified that his criticism wasn’t aimed at his fellow leaders’ offices, and added that Pelosi and Hoyer “don’t deserve to be criticized on hiring practices.” Instead, he said it was the caucus as a whole that needed some diversifying.

“Our staffs are fine, the whole caucus is not fine,” he said Thursday. “All I’m saying is that if you need 90 percent of African American votes, in order to get 50 percent of the total vote, then you ought to allow your staff to be reflective of those congressional districts. That’s all I said.”

Clyburn added that, for him, “diversity means … more than hiring one African-American.”

As reported by Politico, Pelosi and Hoyer didn’t take too kindly to the congressman’s characterization of their office staff and wasted no time highlighting that a majority of their hires are women, folks of color and members of the LGBTQ community.

“Speaker Pelosi’s office is, without rival, the most diverse office on Capitol Hill,” her spokesman, Drew Hammill, told the outlet, adding that: “Speaker Pelosi has always believed in the importance of leading by example.”

Of Pelosi’s personal and leadership staff, 84 percent identify as POC; 48 percent as African-American. Even Clyburn acknowledged that the California congresswoman has “a whole lot of Black folks on her staff.”

Meanwhile, Hoyer’s office boasts a staff that is 70 percent female, people of color or LGBTQ, and 24 percent are African-American, a spokesman told the Journal this week.

A spokeswoman for Clyburn’s office told the Journal that 58 percent of his office is African-American.

The congressman’s most recent remarks come on the heels of two other verbal missteps he’s had to retract. Earlier this month, Clyburn walked back his comments suggesting Democrats were launching an impeachment inquiry against President Donald Trump. Then, in March, he drew the ire of fellow Democrats when he appeared to downplay the significance of the Holocaust.

“Every student of history, which I consider myself to be, recognizes the Holocaust as a unique atrocity which resulted in the deaths of six million Jews,” he said at the time. “It should never be minimized; I never have and I never will.”

Since his latest comments, the South Carolina Democrat has acknowledged Congress’ efforts to address the diversity issue and credited Pelosi and Hoyer with launching the House Democratic Diversity Initiative. Still, he believes there’s more work to be done.

“Many young people of color, however, still feel that our pace is too slow, and I do believe there is still much work that needs to be done to make sure Congressional offices reflect the makeup of Congressional districts and the Democratic electorate,” he said in a statement to Politico.