Secret Service Director Julia Pierson was the first departure in the wake of a string of embarrassing security lapses, but she may not be the last. Speaking Sunday on CBS' "Face the Nation," Representative Elijah Cummings blasted the "culture of complacency" at the Secret Service and warned of further personnel changes to come.

"It did not begin with [Pierson] and it's not going to end with her leaving," said Cummings, the top Democrat on the House Oversight and Government Reform Committee. "There's still people that probably need to go."

Cummings, Representative Jason Chaffetz and other members of the committee lambasted Pierson during a hearing last week over the handling of the president's security, and began digging into a range of institutional problems at the Secret Service. Pierson resigned a day later.

"You had agents that were basically afraid, that the information that they wanted to impart to the top person would never get there," Cummings said. "They were fearful that there would be retaliation."

While he told "Face the Nation" that many African Americans have expressed the view that Obama has not been protected properly because he is black, Cummings rejected that notion.

"We have information that this goes all the way back to the Bush administration," said Cummings, who is also black. "A lot of the problems that we are talking about now, it's just that they are coming to light. A lot of these things existed before our President Obama."

Senior White House adviser Dan Pfeiffer, meanwhile, said that interim Director Joseph Clancy would be charged with weeding out further problems and would be tasked with finding a permanent replacement.

"We'll see who the right fit is," Pfeiffer told NBC's "Meet the Press."