The senior White House staffer’s name has been linked to recent upheaval in the Department of Homeland Security (DHS) as the president seeks more like-minded allies in enforcing immigration law and curbing the flow of migrants into the country. Over the past four days, Trump has withdrawn his nominee to lead Immigration and Customs Enforcement (ICE), announced Nielsen’s resignation and ousted the head of the Secret Service.

However, immigrant rights groups and even some loyal to Trump have expressed concerns that the president, with Miller’s encouragement, will ratchet up his crackdown on illegal immigration, with potentially disastrous results.

“[Trump] is listening and coordinating with a guy who has never had legislative success as a staffer,” a Trump loyalist who advised the campaign told The Hill. “We are going to end up with the same result every time.”

A key voice in the administration’s development and implementation of its travel ban, lowered refugee caps and negotiations over immigration reform, Miller is seen as playing a major role in the administration’s reshuffling of the Department of Homeland Security.

He was reportedly involved in pushing for Nielsen to go, and CNN reported that Miller would like Trump to dismiss U.S. Citizenship and Immigration Services Director Lee Cissna and DHS general counsel John Mitnick as part of broader changes at the agency.

The White House did not respond to questions about Miller’s involvement in Nielsen’s exit or whether he will play a role in picking her successor and did not make Miller available for comment.

Critics of the 33-year-old Miller say he is a huge impediment to any possible compromise bill on immigration reform.

Speaking to reporters Monday morning, deputy press secretary Hogan Gidley chided the media for its interest in the machinations of personnel changes.

“The palace intrigue of how this came about is to me somewhat pointless,” he said.

The reshuffling of top DHS officials comes as the president has pivoted sharply back to immigration in recent weeks. He declared a national emergency to construct a wall, urged those trekking toward the country to “turn around” and threatened to close the U.S.-Mexico border.

On Friday, he withdrew Ronald Vitiello’s nomination to lead the agency tasked with enforcing immigration law.

“We’re going in a little different direction,” he said of his pick to head ICE. “Ron’s a good man. But we’re going in a tougher direction. We want to go in a tougher direction.”

Trump tapped Customs and Border Protection head Kevin McAleenan as the new acting secretary of Homeland Security. The selection of a Border Patrol official to run a sprawling agency that also oversees election security, cyber threats and terrorism prevention underscores the president’s singular focus on immigration.

Groups that support stricter immigration laws were largely supportive of the change in leadership, calling it overdue.

Officials with NumbersUSA and the Center for Immigration Studies suggested Nielsen was not well-versed on the issues of immigration, pointing to her cybersecurity background, and that she was not proactive enough in addressing the flow of migrants at the southern border.

Rosemary Jenks, the director of government relations for NumbersUSA, praised Trump rescinding Vitiello’s nomination and expressed hope that McAleenan’s tenure atop DHS would be short-lived.

Neither man had been effective in their previous role, she asserted, and Nielsen’s replacement should be someone loyal to Trump’s policies and whom law enforcement trusts.

She and Mark Krikorian, executive director of the Center for Immigration Studies, both advocated for Cissna to remain in his job, even as the head of United Citizenship and Immigration Services is reported to be in Miller’s crosshairs.

“That would be maybe the biggest error this administration has made since it was started,” Krikorian said of Cissna’s potential departure.

Krikorian and Jenks view Miller’s influence as one person in a large operation, but noted he serves as an important ally on implementing the president’s immigration agenda.

Krikorian credited him as a staffer who has “actually been fighting to try to move the administration in the right direction on immigration.” Krikorian conceded the senior staffer had “mixed results” thus far.

The adviser to the campaign, meanwhile, said Miller doesn’t seem to have a workable plan to address the immigration system.

“I don’t think the president is happy that the Trump brand has equaled failure on immigration,” the adviser said. “And he has one person to blame for that: Stephen.”

Immigrant rights groups suggested it would be more of the same, or that the Trump administration’s policies could become more extreme, if Miller continues to guide the president.

“Every policy Trump and Stephen Miller want is either impossible, illegal or immoral,” Douglas Rivlin, communication director for America’s Voice, said in a statement. “Their simplistic deterrence-only strategy misreads the complexity of the challenge and will only make the situation worse.”

Nielsen had long been rumored to be on the chopping block. The president reportedly grew frustrated with her that she had not done enough to deter illegal immigration and at times berated her in front of other officials.

While Nielsen stood by Trump’s policies and defended the practice of separating migrant families at the border last year, NBC News reported that her opposition on legal grounds to Trump’s desire to resume the separations contributed to her ouster.

On Friday, Nielsen sat two seats away from the president as he spoke to law enforcement agents at a roundtable in Calexico, Calif. She later walked with him as he toured a section of reinforced border wall.

“This is our new statement: The system is full,” Trump said Friday as Nielsen looked on. “Can’t take you anymore. Whether it’s asylum, whether it’s anything you want, it’s illegal immigration. We can’t take you anymore.”

In her first public comments since the announcement, Nielsen told reporters she would work to ensure a smooth transition prior to her exit on Wednesday.

“I share the president’s goal of securing the border,” she said. “I will continue to support all efforts to address the humanitarian and security crisis on the border, and other than that I’m on my way to keep doing what I can for the next few days.”