At one of the Republican debates last week, Juan Williams of Fox News asked Texas Gov. Rick Perry if federal officials should still be “scrutinizing” state voting laws. Perry’s angry response: Texas is “under assault by the federal government” over voting issues. Perry was objecting to federal investigations into Texas’s voter ID law, which requires voters to have a valid government photo ID. The Justice Department may be about to strike down the law on the ground that it would keep many minority voters from casting ballots.

When the feds blocked South Carolina’s voter ID law from taking effect, Gov. Nikki Haley charged Washington with interfering with “a bill that would protect the integrity of our voting.” South Carolina plans to challenge the decision in court.

The battle over voter ID laws has put a spotlight on the Voting Rights Act, the landmark 1965 law that gives the Justice Department the power to block state voting laws. Increasingly, the fight is shifting from whether voter ID laws should be struck down to whether the Voting Rights Act should be overturned.

It was not long ago that the Voting Rights Act had broad bipartisan support. There was always grumbling about it, especially in the South, but Congress reauthorized it in 2006 by a vote of 98-0 in the Senate and 390-33 in the House. George W. Bush signed it into law. But now more conservatives are taking aim at the act — and specifically, Section 5, which requires states to “pre-clear” changes in election law and policy with the Justice Department. These critics argue that even if this process was necessary in the 1960s, when states like Mississippi were blatantly working to keep blacks from registering and voting, it is not needed now.

Defenders of the Voting Rights Act — and of the Justice Department’s aggressive use of it — argue that minorities still face significant obstacles at the polls. They see voter ID laws as a prime example. The purported reason for these laws is to prevent voter fraud, but there are virtually no documented cases of people going to the polls and pretending to be someone they are not.

What voter ID laws do, however, is make it extra hard for minorities to vote. South Carolina’s own data shows that black voters are 20% more likely to lack acceptable photo ID than white voters. The Brennan Center at NYU Law School found that ID laws could pose serious problems for more than five million eligible voters. If voter ID laws reduce black and Hispanic turnout by a few points, that can make the difference in a close election — particularly since these groups vote heavily Democratic. If the laws change the results in swing states like Florida or Wisconsin, they could change the outcome of the presidential election.

Several challenges to the Voting Rights Act are working their way through the federal courts. Last week, the U.S. Court of Appeals for the D.C. Circuit heard a challenge brought by an Alabama county, which argues that the Constitution does not give Congress the power to second-guess state election laws the way Section 5 does.

There is a good chance this case, or another like it, could soon land in the Supreme Court — perhaps this year, if a voter ID challenge gets fast-tracked so it can be decided before November. There may well be enough votes to hold Section 5 unconstitutional. Rick Hasen, a University of California-Irvine election law expert, has written in his election law blog that the Supreme Court could strike down Section 5 “relatively soon.”

That would be very unfortunate. Even if minorities are not prevented from voting in the same way they were in the 1960s, states and localities are still adopting laws that suppress minority voting. They draw district lines that divide Latinos so they do not get a fair number of representatives. They close polling places in black neighborhoods, making it harder for black voters to cast ballots. They impose onerous rules on voter registration — so onerous, in Florida’s case, that the League of Women Voters has stopped registration drives in the state.

The Voting Rights Act may not be perfect, but it is the main tool we have for ensuring that the powers that be do not put unfair obstacles in the way of voting. By removing unfair rules — like photo ID requirements — it protects the voting rights of white voters, too. Losing the Voting Rights Act would seriously weaken American democracy.