Washington (CNN)President Donald Trump has grown increasingly frustrated with Hope Hicks, his communications director and one of his closest confidantes, amid the fallout from the Rob Porter scandal, people familiar with the matter say.

Meanwhile, the President has told associates he's dismayed at how the allegations involving his former staff secretary accused of domestic abuse were handled, but he isn't certain how to solve the mushrooming controversy, a person familiar with the conversations tells CNN.

Trump was not consulted when Hicks and several other aides drafted a White House statement defending Porter, and he is under the impression that Hicks has let her romantic relationship with Porter cloud her judgment, a source familiar said.

In the aftermath, Trump has told associates he feels that Hicks put her own priorities ahead of his. However, there is little to indicate that Hicks' standing is in jeopardy.

Speaking during the White House briefing on Thursday, spokesman Raj Shah said Hicks had recused herself from some matters related to the Porter fallout. Porter was in the building for a short period to clean out his desk, Shah said.

Hicks continued to privately defend Porter to her White House colleagues Thursday, a source familiar said.

The White House did not respond to questions about Hicks on Friday morning.

Trump is "very disturbed by the negative press attention" the Porter story received, a source close to the President said on Friday.

Trump's senior aides knew for months about the allegations levied against Porter by his ex-wives, even as Porter's stock in the West Wing continued to rise, multiple sources have told CNN. Porter denied the allegations but resigned on Wednesday.

There is growing frustration among Trump's aides and allies in and out of the White House that chief of staff John Kelly badly mishandled the situation against Porter, one of his top deputies. Sources say he knew about some claims that Porter physically and emotionally battered two ex-wives, yet didn't conduct an internal investigation into their veracity.

White House communications director Hope Hicks.

Trump has spent the last two nights phoning friends and former aides to seek their advice, which has varied wildly. Some have told him the episode is a sign he must clean house. Others say drastic moves -- such as firing Kelly -- would only plunge the West Wing into further chaos.

Trump has regularly vented to his associates about Kelly, who imposed a strict system of information flow when he entered the White House over the summer. Trump has chafed at the restricted access, as have some of his regular contacts who no longer enjoyed unfettered access to the President.

Trump has spoken to his former chief of staff Reince Priebus about the current state of affairs in the White House, the person familiar with the conversations said, though not specifically the fallout from the Porter allegations. Priebus told CNN on Friday that he had not discussed the Porter controversy with the President. The two men had lunch at the White House in recent weeks, before the Porter scandal erupted.

While the reason isn't yet clear, Trump also has been quizzing those around him about their opinion of Budget Director Mick Mulvaney in recent weeks. Though he is coy about what he's getting at when he asks about Mulvaney, most aides and associates assume he is referencing the chief of staff position. Mulvaney's profile rose during the government funding talks.