At the Supreme Court argument Wednesday on the Voting Rights Act, Justice Anthony Kennedy – likely the deciding vote – had two potentially devastating words for those who want to see the landmark voting rights law upheld: “Times change.” It can be perilous to try to predict what the court will do based on the questions the Justices ask at oral argument. But those questions suggest that there may well be five votes — a majority — for striking down key parts of the act.

The Voting Rights Act was enacted in 1965 to ensure that black voters in the Jim Crow South were allowed to cast ballots. In the 48 years since, it has been a tool for the federal government to prevent states from racial gerrymandering — drawing district lines to stop minorities from getting elected — and election-day obstructions, like moving polling places at the last minute in minority neighborhoods.

Conservatives have long been at war with the Voting Rights Act. They argue that it gives an unfair preference to minority voters, infringes on states’ rights, and is an abuse of power by Congress. The case the court is considering, Shelby County v. Holder, challenges a key part of the act: section 5, which requires all or part of 16 states to “pre-clear” changes with the Justice Department to ensure that they do not unfairly burden minority voters.

Going into the argument, the court’s four most conservative Justices were all-but-certain votes against section 5, and that seems just as true now. At the argument, Chief Justice John Roberts — who has expressed skepticism about the Voting Rights Act for decades — took up the line being pushed by Shelby County, Alabama: that the act unfairly puts a heavier burden on southern states (even though it also covers some northern jurisdictions). Chief Justice Roberts asked: “Is it the government’s submission that the citizens in the South are more racist than the citizens in the North?”

Justice Antonin Scalia — as is his wont — was less subtle. He called the act “a perpetuation of racial entitlement.” Justice Alito echoed Chief Justice Robert’s concerns, asking why Congress did not make a “new determination” about which states and localities should be covered. Justice Clarence Thomas did not ask any questions — his longstanding practice — but four years ago he voted to strike down the act, and it is all but certain he will again.

The seeming solidity of those four conservative votes makes Justice Kennedy’s skeptical stance potentially decisive. At the argument, he said he saw the Voting Rights Act as “utterly necessary” in 1965, but it was now “not clear” to him. He added: “The Marshall Plan was very good, too — the Northwest Ordinance, the Morrill Act – but times change.”

The court’s four more liberal Justices used their questioning to underscore the reasons why the act is still needed – and why Congress had ample constitutional authority to enact it. Justice Sonia Sotomayor told the lawyer for Shelby County that even if parts of the South have changed “your county pretty much hasn’t.” Justice Stephen Breyer compared the racial discrimination that created the need for the act to a disease, and said that things may have changed to some degree “But we know one thing: the disease is still there in the state.” All these four liberal Justices can do without a fifth on their side, however, is write a dissent.

Cases do not always come out the way they appear headed in oral argument. Justice Kennedy, or one of the other conservative Justices, may ultimately balk at striking down a key part of a revered civil rights law that has been enacted repeatedly by Congress with bipartisan support. In the end, the court may use this case to give the Voting Rights Act a ringing endorsement. But right now, that is probably not the smart bet.