Republican members of Congress are abandoning their pledge to never raise taxes, but Grover Norquist says he isn’t worried. The handful of Hill Republicans who have gone wobbly were never reliable and are no threat to the overall project of shrinking government through lower revenue, says Norquist, head of Americans for Tax Reform and the pledge’s author and driving force. “We’re fine,” he tells TIME. “Those are the same guys you were quoting to me two years ago.”

That’s only partly true. Over the weekend, a handful of Republicans said they would consider abandoning the pledge. Senator Saxby Chambliss of Georgia told a local TV station, “I care more about my country than I do about a 20-year-old pledge.” Lindsey Graham of South Carolina said bluntly that he would break the part of the pledge that protects tax shelters for the rich. Several other Republicans, including Norquist’s archenemy, Senator John McCain, as well as Representative Peter King and Senator Bob Corker, suggested they too might back away from the commitment.

The GOP retreat has raised questions about whether Norquist’s pledge, long a sacred creed for Republicans in Washington, is crumbling. But he is quick to argue that these members of Congress were never true believers. Chambliss “is in a safe district and can say what he wants,” says Norquist. As for Graham, Norquist says he was wobbly two years ago as well and that journalists are just rediscovering an old story.

More worrying for Norquist are the mixed signals being sent by Eric Cantor, the powerful House majority leader who spearheaded resistance to a grand bargain in 2011. On Monday morning, Cantor told MSNBC that pragmatism was more important than the pledge. “When I go to the constituents that have elected, re-elected me, it is not about that pledge,” he said. “It is really about trying to solve problems.”

Norquist says he talks to GOP leaders in the House and that “they don’t intend to raise taxes and they’re looking to get a good deal.” He says he doesn’t think the leaders intend to violate the pledge on deductions, which requires spending cuts for any shrinking of tax loopholes. Republicans and Democrats have recently focused on closing loopholes as a potential way around the pledge.

The pledge is perhaps the most succinct encapsulation of the hard-right fiscal approach of a party that just lost a presidential election. The view holds that government is the problem (for liberty, the economy, society, you name it), and it is best constrained by cutting off its funding, i.e. taxes. Norquist famously said, “I’m not in favor of abolishing the government. I just want to shrink it down to the size where we can drown it in the bathtub.” And he believes unreservedly in the supply-side idea that lower taxes equals greater growth.

Norquist’s sharp tongue and decades-long focus on tax policy make him a target for Democrats who say he has an unnatural stranglehold on the GOP. But it’s the potency of the raising-taxes issue with voters that members of Congress fear, not the diminutive ideologue. You don’t get support from virtually every member of a major political party through the force of your argument in the U.S.; you get it because it’s a winning issue with voters.

Even if Norquist’s pledge is weakened by a grand bargain, he’s likely to bounce back. He survived a much more damaging episode in 2005 when he acted as a pass-through between his old friends Jack Abramoff, the disgraced lobbyist, and Ralph Reed, the conservative Christian leader, in their efforts to block a state lottery that would have competed with the interests of one of Abramoff’s clients.