Close allies of Speaker John Boehner are worried that his conservative rivals could move to oust him as soon as next week.

Removing a sitting speaker is exceedingly difficult, and such an effort would almost certainly fall short. Yet growing speculation about the possibility of it – coming after Friday’s embarrassing defeat at the hands of conservatives and House Democrats on the homeland security battle — shows how vulnerable the speaker has become.

Story Continued Below

Five years into the job, he’s a leader consistently buffeted by forces beyond his control. The legislative calendar guarantees it won’t get any easier: in the coming weeks and months there will be battles over the debt ceiling, budget, taxes, and spending cuts.

The question is how many more of these episodes Boehner can withstand.

Frustration with the Ohio Republican is mounting after dozens of hardliners voted Friday against his three-week funding package for the Department of Homeland Security. Hours of frantic leadership meeting ensued. After some backroom maneuvering with Democrats, Boehner was able to push through a one-week bill to keep DHS open.

President Barack Obama signed the bill into law just 10 minutes before a shutdown of the massive federal agency.

The stinging rebuke of Boehner on the House floor infuriated his supporters, who accused opponents of handing Democrats a huge PR victory.

Yet is also left even Boehner backers wondering how viable he remains. They admit these repeated confrontations, in which Boehner can’t muster 218 Republican votes for his proposals and has to turn to Democrats for help, leave him looking weak and ineffective — and thus vulnerable to a conservative challenge.

“Some of these 52 [Republican who voted no] are more worried about protecting their own careers than protecting their constituents from ISIL. They are more worried about primaries than they are about the country,” said a GOP lawmaker close to Boehner, using an acronym for the Islamic State. “This is all aimed at Boehner. They want to take Boehner out.”

While there’s no open movement afoot to replace Boehner, his allies are plainly nervous that GOP hardliners might try.

Members of the recently formed “House Freedom Caucus” offered multiple proposals to leadership that they believe would have drawn enough Republican votes to keep DHS funded and not left Boehner dependent on Democrats. Boehner, though, chose not to support the plans.

One conservative member, who asked for anonymity to speak frankly, said the mood of his colleagues will depend on how Boehner handles himself over the next week. If he tries to put a “clean” DHS funding bill on the floor for a vote, or doesn’t make overtures to conservatives, anger could boil over, the Republican said.

The lawmaker said he will be watching closely the three dozen or so members who voted for Boehner during the speaker election in January but who have been critical of the GOP leadership team’s tactics.

A handful of Boehner loyalists met with him Friday night and voiced concern about behind-the-scenes agitation among the most conservative faction of the conference.

The chances that an attempt to remove him would succeed are virtually nil. But it would show that his critics are willing to risk a split within the GOP Conference to force him out.

How Boehner would respond to such an effort is anyone’s guess. It’s seen as highly unlikely he would step down. Yet he’s also shown no inclination to seek a showdown with his GOP critics or move against them by taking away committee assignments, cutting off campaign money, or endorsing primary challengers. Such tactics might actually help the rebellious lawmakers by boosting their profile with conservative groups, which despise Boehner.

All of which leaves the House GOP stuck in its current rut: run by a weak speaker, lurching from crisis to crisis, and burdened with an internal opposition that has a strong ideological position but no actual plan to govern.

Unhappiness with Boehner has been growing since the end of last year. He decided then to move forward with the so-called “cromnibus” that funded the government through September, except for DHS, which received money only through Friday. The idea at the time was to avoid another damaging government shutdown — and then use homeland security financing as leverage to force President Barack Obama to back away from his actions to shield millions of illegal immigrants from deportation.

Instead, conservatives believe Boehner left them with less leverage to scuttle the president’s immigration policies.

But it’s one thing to disapprove of Boehner’s leadership. Booting him is another matter.

For one, the vast majority of House Republicans like and support him. And there’s no obvious replacement to rally around.

Ways and Means Committee Chairman Paul Ryan (R-Wis.) is seen as a potential speaker, but he’s fiercely loyal to Boehner. House Majority Leader Kevin McCarthy (R-Calif.) has close ties to many rank-and-file members, but he’s also a Boehner loyalist.

Names aside, any new speaker would have to contend with the same political forces that have hemmed in Boehner. There’s no evidence someone else could do any better.

So far, the 65-year-old Boehner, who was first elected to the House in 1990, is not taking the possibility of a coup attempt seriously, GOP sources said.

“It would take a big uprising to do something and right now, I mean, I think really, truly Boehner is a stabilizing force,” said Rep. Lynn Westmoreland (R-Ga.), who is close to Boehner. “We’ll see what happens. Next Friday will be a big day. Next week will be a big week. We’ll see what he can do.”

Boehner was asked at a Thursday news conference whether his speakership is on the line. “No! Heaven’s sake, no,” he said. “Not at all.”

Some lawmakers are hopeful the homeland security debacle will provide a much-needed moment of clarity.

Rep. Frank Lucas (R-Okla.) said the question is “who is going to run the House. Is it going to be the Democratic minority? Is it going to be a very focused minority within the House Republican Conference? Maybe this is one of those points in time where we have to sit down and figure out which way we want to go. But you can’t have the tail on both ends wagging the dog in the middle.”

Though Lucas remains supportive of Boehner, he was cautious about evaluating the speaker’s future.

“He is the speaker of the United States House until he’s not,” Lucas said.

To remove Boehner from the speaker’s chair, a lawmaker would introduce a “motion to vacate,” which an overwhelming majority of GOP lawmakers would have to back. Republicans say that would never happen in the conference right now.

Conservatives said immediately after the vote late Friday that they have no plans to challenge Boehner despite all the behind-the-scenes talk of forcing him out.

“I’ve had my differences with the speaker at times both on tactics and policy,” said Rep. John Fleming of Louisiana. “But we elect each speaker for two years. There is no discussion or talk among conservatives to get him out.”

Conservative South Carolina Republican Rep. Trey Gowdy was also measured in his criticism of Boehner.

“Being in leadership is a tough job, which is why so few people raise their hands and volunteer to do it. It’s easy where I sit just to kind of second guess,” said Gowdy, who voted against the one-week funding bill after earlier supporting the three-week plan. “I believe in self-reflection, and then after that self-reflection, if you have something to say, you say it to them personally.”

But impatience is rising among conservatives.

“I think we are frustrated that they continue to reach out to the Democrats of the Senate instead of working with conservatives,” said Rep. Mick Mulvaney.

When pressed by reporters whether conservatives would move to expel Boehner, however, the South Carolina Republican said he hasn’t heard “any discussions like that with anybody.”

Hardliners may have missed their best chance to remove Boehner: the vote for speaker on Jan. 6. Twenty-five Republicans voted against Boehner that day, an extraordinary show of opposition to a sitting speaker, especially one who a few months earlier had led his party to their biggest majority in decades.

In fact, Boehner was fortunate that several New York Democrats were missing that day for the funeral of the late Gov. Mario Cuomo. That allowed him to win reelection as speaker with only 216 votes.

Since then, some of Boehner’s allies have been clamoring for retribution against the rebels. But Boehner won’t do it.

Rep. Mike Simpson (R-Idaho), one of Boehner’s closest allies, dismissed all the speculation about ousting Boehner as meaningless.

“It’s more talk,” Simpson insisted, “than anything else.”

Correction: An earlier version of this story misstated Lucas’ home state.