FOR ALL THE controversies that trigger debate or polarize the kingdom of sports, this was one that offered little in the way of ambiguity: There was a right side and a wrong side, with scarce space in between. There's no debating the offensive nature of Clippers owner Donald Sterling's comments in the recording that mysteriously surfaced of an argument he had with his acquaintance/archivist/self-proclaimed "silly rabbit," V. Stiviano. The shock value of Sterling's racist ranting was high, and the reaction of both the public and NBA commissioner Adam Silver was swift. Sterling stands to become the first NBA owner to lose his team over bigoted speech.

But there is plenty of legal ambiguity in the punishment. Silver fined Sterling $2.5 million, permanently banned him from the NBA and recommended that NBA owners vote to force the 80-year-old real estate magnate to sell. This all raises serious concerns about power, process and precedent.

Take the swiftness of NBA justice. In merely four days Silver began and completed an investigation into a recording that may have been illegally created (California requires both parties to consent), analyzed the NBA's bylaws and constitution, and issued a sweeping, unprecedented penalty. For a league that is famously deliberate and cautious—it has taken years to decide if jerseys should be adorned with advertisements—four days to ban the longest-tenured owner was awfully quick, which could give Sterling legal standing.

Yes, Silver faced overwhelming pressure to respond decisively. Players threatened to boycott playoff games. Sponsor after sponsor dropped the Clippers. President Obama linked Sterling's "incredibly offensive, racist statements" to "the legacy of race and slavery and segregation." But when a punishment is as much about quelling outrage as it is about achieving justice, questions can be raised about whether fairness is being compromised. To some—Sterling's army of lawyers, especially—the Clippers' owner may look like like a victim of frontier justice. In other corners his punishment is a matter of a private industry acting with admirable efficiency. That debate will be settled by a legal system that can be slow and deliberate but is designed to offer consistency and fair recourse even to racist billionaires.

To be sure, the NBA's legal process is unlike that found in courts. Silver is afforded broad discretion to punish owners, and Sterling agreed to this system when he bought the team in 1981 and joined one of the nation's most exclusive clubs. The NBA would likely prevail against a legal challenge. But potential litigation could take years, and Sterling could raise powerful arguments.

For starters, Sterling could insist that he did not "willfully" break any rules. This is important because the NBA clause that best justifies an expulsion requires willful conduct. Sterling was embroiled in a heated argument over a photo of Stiviano with another man. While Sterling made disparaging remarks about African-Americans' attending Clippers games, the Clippers have had multiple African-American coaches (including Doc Rivers, who signed a $21 million deal last summer), which might even count in Sterling's favor.

Even in the unlikely event that the NBA can establish that Sterling intended to skirt the league's bylaws, it's not clear such an extraordinary penalty is justified. Sterling wasn't convicted of a crime and doesn't stand accused of failing to meet his financial commitments. He wasn't charged, arrested or indicted. He wasn't investigated by the police or the FBI. He can't even be sued for his remarks. One might expect that the first time in the league's 68-year history that an owner is on the verge of expulsion would involve some kind of unlawful conduct. Ironically, any unlawful conduct in the Sterling scandal would have been committed by others against him.

Silver asserts that Sterling's troubled history can help to justify his ouster. Should it? Until last week the NBA had practiced a hear-no-evil, see-no-evil approach to Sterling's prior bad acts and broken promises. For decades he happily pocketed league TV money while refusing to put a competitive product on the floor. Worse yet, the NBA ignored accusations of racism in a real estate case brought by the U.S. Department of Justice and in a lawsuit by former general manager Elgin Baylor. Silver stressed that the league could not act in those instances because the cases were either settled or won by Sterling. But now Sterling faces banishment for words that can't even give rise to a lawsuit?

What are Sterling's legal options? He could seek an injunction to temporarily halt his ouster by asserting that he would suffer irreparable harm by losing an NBA franchise and has at least a plausible case in court. He could also attempt to transfer his Clippers ownership stake to his family—a maneuver the NBA could challenge, but in doing so would risk a lawsuit by family members. Sterling's estranged wife, Rochelle, could file for divorce and stake a claim in her husband's assets—including his equity in the Clippers. A family judge might then delay the NBA's ouster of Sterling.

Sterling's most powerful weapon may be to file a lawsuit arguing that the NBA has misapplied its own rules and conspired against him. Neither of those claims would most likely prevail because the standard of review would be more deferential to the NBA, but both could enable Sterling's lawyers to depose owners under oath. Sterling would seek testimony to prove that owners who are prepared to vote him out for racism are guilty of comparably objectionable behavior. Sterling would demand that former commissioner Stern testify about his knowledge of owners' transgressions. It could get ugly, especially if Sterling knows where the skeletons are buried.

Despite his infamy, Sterling isn't the only sports owner to exhibit questionable viewpoints and conduct. A company controlled by Astros owner Jim Crane, for example, has reportedly paid millions to settle discrimination claims. Crane has been accused of warning managers, "Once you hire blacks, you can never fire them." Browns owner Jimmy Haslam, who owns a nationwide string of truck stops, is under federal investigation for fraud. The DeVos family owns the Magic, and senior chairman Rich DeVos openly criticizes marriage equality—despite the NBA's antidiscrimination goals—while Mavericks owner Mark Cuban was sued (unsuccessfully) by the Securities and Exchange Commission.

In the future, should owners who draw the wrath and outrage of the public be subject to the loss of their franchises? Regardless of how you feel about Sterling's behavior, a precedent has now been established. Maybe it shouldn't have been.

Litigation could take years, and Sterling could raise powerful arguments.