"I can say it is the defining moment for this speaker. If he gets it wrong it will have consequences for him, but it will also have consequences for the rest of the Republican Party," Meadows said at an event with fellow House conservatives on Capitol Hill on Wednesday.

The North Carolina Republican stressed, "this president was elected largely on an immigration issue that defined him differently than the other candidates and so it is the defining moment, more so than the budget or anything else that we've passed."

Ryan's office did not provide a response to Meadow's comments.

Ohio GOP Rep. Jim Jordan also piled on Ryan, a Wisconsin Republican. He criticized Ryan for backing the budget deal that avoided a government shutdown last week, tweaking the speaker, who rose to national prominence for his focus on cutting deficits.

Read More

"Just a few years ago Speaker Ryan was viewed as the -- the -- individual in our party, in our country, who was focused on fiscal responsibility, and then you saw what happened last week."

Jordan also pointed out that Ryan's positions on immigration do not sync up with many House conservatives. "So all I'm saying is we need legislation in every policy area that is consistent with that we told them we were going to do and what they elected us to do. That did not happen last week, but I certainly hope it happens on immigration," Jordan said.

House conservatives are demanding a vote on an immigration proposal from House Judiciary Committee Chairman Bob Goodlatte, R-Virginia, and argued that the Republican-controlled House should ignore any deal that could emerge from the current Senate debate.

House GOP Whip Steve Scalise and his team are in the process of determining whether that plan has enough support to pass, according to multiple Republican sources. But a bloc of moderate Republicans have expressed strong concerns with it, and Democrats have indicated it's a non-starter so it's unclear that the plan as drafted could clear the chamber.

Ryan has been careful to say that House Republicans would continue to work on a proposal separate from the Senate, and that he would move a proposal that President Donald Trump backed, and he reiterated that position Wednesday morning.

"What we always want to do in the House is preserve a House Republican position. So which we can start from for negotiations, but at the end of the day we want to have a solution," Ryan told reporters on Capitol Hill.

The speaker made it clear he was more concerned with the White House position than the Senate process, "we'll see what the Senate does this week. Frankly I have no idea what the Senate is going to produce this week and we'll also have the president engaged to make sure we have something that if it lands on his desk he's going to sign it, and that's very important to us."

A new speaker?

Meadows qualified that there weren't any conversations "right now" about a new speaker but added "there are certainly conversations that would involve new leadership." He stressed that the difference was that he was awaiting Ryan's next move on immigration, saying, "we need leadership to actually lead."

Multiple House GOP members did not hide their frustration that the Senate is moving forward on an immigration debate while their own leadership team is still figuring out a House strategy.

"This is a watershed moment for all of us," South Carolina GOP Rep. Ralph Norman said about the immigration debate. "This was the 2016 election."

Meadows added, "anytime that the Senate can take up an immigration bill quicker than the House is a day that should defy history. I mean the Senate is notorious for taking naps and not taking votes and yet we somehow are here in situation where the Senate is actually beating us to the critical issue of the day on immigration. That shouldn't have happened."

Republican Rep. Scott Perry of Pennsylvania told reporters that his leadership needed to be "as invigorated" about the conservative House immigration proposal as they were about other items they worked over GOP members to vote for such as Obamacare repeal, tax reform and the budget proposal.

"But if it's doomed to fail right from the beginning, then we are going to have problems," he said flatly.