The IRS is unpopular on its best days, and the past few have been among its worst. The agency’s admission that it targeted conservative groups for special scrutiny drew condemnation from across the political spectrum on Monday. “Outrageous,” declared Barack Obama. House and Senate leaders from both parties promised an investigation. Some of the Tea Party groups refused to even accept its apology.

All this outrage threatens to obscure an important point: the IRS does need to crack down on political groups masquerading as social-welfare organizations. Many of the nonprofit groups who claim 501(c)(4) status either flout tax law or flirt with the murky line between electioneering and issue advocacy, all while using their tax-exempt status to conceal their donors. The problem isn’t that the IRS flagged nonprofit groups for additional review. The problem is that it did so poorly, lavishing special attention on Tea Party outfits when it should have been scrutinizing everyone — or at least more egregious offenders.

This is easier said than done. After the U.S. Supreme Court’s Citizens United decision in January 2010, donors flocked to 501(c)(4)s as a vehicle to pump cash into elections without disclosing the source of their contributions. The number of groups applying for social-welfare status has since doubled. In 2012, the news outlet ProPublicaexamined 72 501(c)(4) applications from groups that claimed to have no plans to spend money on elections. They compared those documents against the subsequent tax returns. Nearly half of the groups found their plans had changed.

In last year’s elections, 501(c)(4) groups spent more than $300 million in dark money, according to Lisa Rosenberg of the Sunlight Foundation, a nonpartisan government-transparency group based in Washington. There is no way to police all these groups, Rosenberg acknowledges. But the IRS, deluged with social-welfare applications at the same time the Tea Party movement was on the rise, appears to have picked a political filter as a shortcut. “It’s the right thing to do to be looking into which of these groups are legitimate social-welfare organizations and which are political organizations. It’s absolutely necessary,” Rosenberg says. “There’s no question the way the IRS apparently went about it was wrong. But the fact that they were doing it is absolutely right.”

The method the IRS used to determine which groups to investigate — singling out keywords like tea party, patriot and other conservative terms of art — was “just backwards,” says Fred Wertheimer, president of the campaign-finance watchdog Democracy 21. “There are a number of groups that have blatantly been abusing the tax laws in order to hide their donors. Those are the groups that the IRS should have been investigating.”

Beginning in the fall of 2010, Democracy 21 and another nonpartisan group called the Campaign Legal Center have urged the IRS to crack down on groups that have improperly claimed social-welfare status. Among the leading offenders, Wertheimer says, are two right-leaning groups, Crossroads GPS and the American Action Network; PrioritiesUSA, which supported Obama’s re-election; and the short-lived Americans Elect, which tried to raise a third-party presidential candidate to compete in 2012 despite registering as a social-welfare nonprofit.

The IRS, Wertheimer says, was provided with evidence that clearly documented how these groups flouted IRS regulations, which hold that they must be “primarily” engaged in social-welfare activities rather than electioneering. (Crossroads GPS founder Karl Rove even acknowledged in a Wall Street Journal editorial that his group had funneled millions into an ad blitz on behalf of Mitt Romney.) And yet the IRS chose instead to adopt a crude criteria targeting possible Tea Party transgressors.

One of the predictable ironies of the IRS decision is that the scandal has breathed new life into the groups the organization had sought to disrupt. Until last week, the Tea Party had been on the wane. Now it is back in the news, fed by outrage at its persecution. “They came up with a formulaic way of looking at groups simply because of their names,” Wertheimer says of the IRS. “Which was wrong. And they haven’t dealt with the real abuses that are going on.”

For campaign-finance watchdogs, the fear is that the backlash will spook the IRS out of pursuing major players who are using the loophole to influence elections. “Our concern now,” Wertheimer says, “is to make sure the focus on IRS abuses does not become cover for political organizations that are blatantly misusing the tax laws.”