WASHINGTON — In his six months as Homeland Security secretary, John F. Kelly often described the White House as one of the most dysfunctional organizations he had ever seen, complained to colleagues and allies about its meddling, incompetence and recklessness, and was once so angry he briefly considered quitting.

Now as President Trump’s chief of staff, he is doing something about it — with a suddenness and force that have upended the West Wing.

Mr. Kelly cuts off rambling advisers midsentence. He listens in on conversations between cabinet secretaries and the president. He has booted lingering staff members out of high-level meetings, and ordered the doors of the Oval Office closed to discourage strays. He fired Anthony Scaramucci, the bombastic New Yorker who was briefly the communications director, and has demanded that even Mr. Trump’s family, including his daughter Ivanka Trump and son-in-law, Jared Kushner, check with him if they want face time with the president.

On Wednesday, his third day on the job, he delivered a message about respecting chains of command, backing the decision of Lt. Gen. H. R. McMaster, the national security adviser, to dismiss Ezra Cohen-Watnick, a Kushner ally and staff member on the National Security Council. It was a move Mr. Kushner and Stephen K. Bannon, the president’s chief strategist, had long opposed, according to two administration officials.

Whether Mr. Kelly, a retired Marine general, will succeed in imposing military discipline on the faction-ridden White House remains in doubt; Mr. Trump has never been known to follow anybody’s direction, in Trump Tower or the White House. But Mr. Trump has never encountered anyone quite like Mr. Kelly, a combat veteran whose forceful management style and volatile temper are a match for the president’s.

“He’d basically look at me and say, ‘I think that proposal is four-letter-word nuts,’” said Leon E. Panetta, who as defense secretary made Mr. Kelly his chief military aide. “John is the kind of guy who will look you in the eye and tell you what the hell he is thinking. The real question is whether the president will give him the authority he needs to do the job.”

People close to Mr. Kelly said he resisted weeks of entreaties by the president, beginning in May, before finally agreeing to replace Reince Priebus out of a sense of soldierly duty. That he understands the sobering realities of his new deployment could be seen in his unsmiling mien while sitting next to Mr. Trump for a photo opportunity this week.

Among Mr. Kelly’s immediate challenges: brokering peace between warring factions in the West Wing; plugging leaks about internal activities; establishing a disciplined policy-making process; and walling off the Russia investigations.

Mr. Kelly, 67, has told his new employees that he was hired to manage the staff, not the president. He will not try to change Mr. Trump’s Twitter or TV-watching habits. But he has also said he wants to closely monitor the information the president consumes, quickly counter dubious news stories with verified facts, and limit the posse of people urging Mr. Trump to tweet something they feel passionately about.

He has privately acknowledged that he cannot control the president and that his authority would be undermined if he tried and failed. Instead, he is intent on cosseting Mr. Trump with bureaucratic competence and forcing staff members to keep to their lanes, a challenge in an administration defined by tribal loyalties to power players like Mr. Kushner and Mr. Bannon.

“Several times I’ve been on phone conversations with the president over the last couple of days and General Kelly has been on those conversations as well,” Mick Mulvaney, the director of the Office of Management and Budget, told reporters on Thursday when asked if Mr. Kelly was making a mark.

The Trump White House is a judge-a-book-by-its-cover workplace, and staff members have been struck by Mr. Kelly’s bearing: tall, stern and commanding a respect Mr. Priebus never did. People close to Mr. Kelly said they expected him to methodically assess his new staff before making more drastic changes — and he has told people that he wants to improve morale before attacking the organizational chart.

Mr. Kelly has not been shy about letting Mr. Trump’s staff members know when they screwed up, ripping into West Wing aides during the chaos surrounding the president’s original travel ban when he was at the Department of Homeland Security. While he supported the broad policy goals, he was furious that he and his sprawling agency’s staff were caught off guard by a directive that was conceived and carried out by inexperienced aides in the White House, according to several longtime Trump advisers.

People close to Mr. Kelly said he also bristled repeatedly at efforts by Mr. Bannon and Stephen Miller, the president’s senior adviser, to install people they liked in his department. Mr. Kelly eventually won pitched battles over who would become director of Customs and Border Protection and head of the Secret Service, officials said. But Mr. Bannon has had a longstanding alliance with Mr. Kelly, supporting many of his other appointments.

In May, Mr. Kelly considered resigning after Mr. Trump’s firing of James B. Comey, the F.B.I. director, telling Mr. Comey in a phone call that he was thinking about doing so to protest the president’s actions, according to a former law enforcement official familiar with the conversation.

A senior White House official briefed on the exchange by Mr. Kelly said he never threatened to quit, but confirmed that he called Mr. Comey.

Days later, Mr. Kelly objected strenuously to the decision by Thomas P. Bossert, Mr. Trump’s Homeland Security adviser in the White House, to take control of the response to a global cyberattack — a role traditionally played by Mr. Kelly and his department’s cybersecurity division.

On Capitol Hill, Mr. Kelly is viewed with a mix of admiration for his long military service and disappointment that he has been too willing to embrace and defend Mr. Trump’s more controversial policies, especially on illegal immigration.

In closed-door meetings with House members in March, Democrats questioned Mr. Kelly about aggressive immigration sweeps at churches and hospitals. The frustration grew as Mr. Kelly disputed that such sweeps were happening, even in the face of enlarged photos showing a Homeland Security vehicle parked on the grounds of Christ Cathedral in Garden Grove, Calif.

“He’ll push back hard,” said Representative Lou Correa, Democrat of California, who presented the photographic evidence to Mr. Kelly during the meeting.

The next month, Mr. Kelly offered a taste of his blunt approach, telling lawmakers they could “shut up” if they did not like the laws his department was charged with enforcing.

“He’s never come to Capitol Hill and blown smoke to senators and congressmen,” said Senator Tom Cotton, an Arkansas Republican and former Army officer who is close to Mr. Kelly.

Dealing with Mr. Trump’s family, especially Mr. Kushner, will not be so simple.

In an interview in May, Mr. Kelly came to the defense of the president’s son-in-law, who has an office in the West Wing as a White House adviser, against charges that he had tried to set up an inappropriate communications channel with Russia. He called Mr. Kushner “a great guy, a decent guy.”

In discussions with Mr. Trump about moving to the White House, Mr. Kelly also insisted that Mr. Kushner and Ms. Trump, who is also a White House adviser, report to him. They both agreed, in part because they wanted to see Mr. Priebus ejected as quickly as possible, and in part because they recognized Mr. Trump’s presidency needed to be professionalized.

A lingering personnel question gave Mr. Kelly a chance to assert his position at the top of the West Wing. Aides said the ouster of Mr. Cohen-Watnick was intended as a show of confidence from Mr. Kelly to Mr. McMaster. Mr. Kushner did not object to the decision, and had conceded that Mr. McMaster was going to fire his friend three weeks ago, according to a person close to the Trump family.

Robert M. Gates, the former defense secretary, who has known Mr. Kelly for two decades, said the fact that the president agreed to have family members report to the new chief of staff was “a really important first step.”

“The question is, does it last?” he added. “But it sends a powerful signal to the rest of the people in the White House.”

Mr. Gates, who was also Mr. Kelly’s boss as defense secretary, recalled the times he sat with Mr. Kelly at the Pentagon across a small conference table once used by Jefferson Davis when he was secretary of war. Mr. Gates would tell Mr. Kelly what he was planning to do and Mr. Kelly would say, “You could do it that way.”