When it comes to hatred and discrimination, white supremacists and neo-Nazis stand in a class by themselves. But the public condemnation heaped on far-right nationalists, and on President Donald Trump for pandering to them, should not be reserved just for the nation’s most blatant racists.

For most Americans, it’s instinctual to reject those who wave Ku Klux Klan–style torches or banners bearing swastikas. But what about the racists who boast law degrees and sparkling resumes, and sport suits and ties? These haters, too, have flourished in the Trump administration. And their policies of bigotry, safely cloaked behind mainstream-sounding think tanks and federal commissions, can do as much or more damage as the thugs on the street.

This is particularly true in the arena of voting rights, which lies at the heart of recent clashes over Confederate monuments that were, for the most part, built at the height of the Jim Crow era. The poll taxes and literacy tests of that period may be gone, but they’ve been replaced by GOP-authored voting restrictions, such as voter-ID laws and barriers to registration, that disenfranchise African American and Latino voters.

President Trump isn’t the first Republican to bandy about unsubstantiated claims of widespread voter fraud, or to use racial code words to rally white voters. But Trump’s call before the election for his supporters to monitor the polls in “certain areas,” and his baseless claims following Election Day that three million to five million noncitizens had voted illegally, took the GOP campaign to intimidate and challenge nonwhite voters to a new level.

Most importantly, Trump’s “election integrity” commission, established in May and dominated by conservatives who have built their careers on promoting voter-fraud myths and ballot restrictions, may set the stage for sweeping new national restrictions on registration and voting. The commission’s de facto head, Kansas Secretary of State Kris Kobach, has pushed to require voters in his state to show proof of citizenship just to register.

Another commission member, J. Christian Adams, is the president of a group that has repeatedly threatened legal action against local election officials, typically in jurisdictions dominated by African American or Latino voters, if they don’t purge their voter rolls. That group, the Public Interest Legal Foundation, is just one of several conservative organizations devoted to imposing new restrictions on voters in the name of battling supposed voter fraud—which numerous credible studies have concluded is virtually nonexistent.

The best-funded of these groups (with a budget of $4.6 million in fiscal 2015) is the so-called American Civil Rights Union, which also uses lawsuits as a means to force voter-roll purges in minority-dominated voting jurisdictions. There’s also True the Vote, which announced plans in 2011 to recruit one million citizens to serve as poll watchers in 2012, and which was investigated by the Justice Department following allegations of voter intimidation.

The newest such group on the scene, run by former Trump campaign aides and dubbed Look Ahead America, has announced plans to block “fraudulent” votes by deploying poll watchers carrying video cameras. (The group’s main purpose, its organizers say, is to register new conservative voters.)

The conservatives leading the charge against supposed voter fraud cast themselves as patriots protecting the integrity of the vote. They are not on the streets swinging clubs alongside neo-Nazis and white nationalists, and they use legal briefs and positions papers, not violence, to achieve their aims. But a string of court rulings have found their voter restrictions in Texas, North Carolina, and elsewhere are both unconstitutional and explicitly discriminatory.

Last year, a federal appeals court rejected a package of North Carolina voting laws, enacted in 2013, on the grounds that many of its restrictions targeted African American voters “with almost surgical precision,” a finding upheld by the U.S. Supreme Court. The year the omnibus bill was enacted, conservative activist and GOP precinct chairman Don Yelton told The Daily Show that it was “going to kick the Democrats in the butt,” and that if it “hurts a bunch of lazy blacks that want the government to give them everything, so be it.”

Conservatives who champion voting restrictions may not have the same motives as white nationalists, but the outcome of their actions, from the perspective of targeted groups, is the same, says Wendy Weiser, director of the Democracy Program at the Brennan Center for Justice.

“The aims of vote suppression, in fact, reinforce the aims of the white supremacist groups,” says Weiser. “And if folks claim to oppose the white supremacist groups, they should also oppose the agenda of those groups and efforts to undermine African American and minority political power.”

After all, the Klan first arose during Reconstruction to stop blacks from participating in politics—in particular, from voting. Sound familiar?