Years ago, I proudly sat in Wrigley Field with my youngest son Aidan for his first visit to my hometown team, the Chicago Cubs. There were not a lot of people in the stands due to the rain, but Aidan and I still cheered the Cubs as they lost to the Reds. A few seats behind us were two obnoxious drunks from out of town who yelled insults throughout the game at players.

Then, in the seventh inning, they screamed “F—k you Barney” as infielder Darwin Barney went to bat. One of the coaches immediately popped up in front of the Cubs dugout and yelled, “Hey, no swearing. Keep it clean.” I immediately voiced my agreement. In truth, neither of our objections had the same impact as the four large (and fairly lobbed) guys in front of us who stood up in Cubs jerseys and said, “Yeah, no swearing at Wrigley.” Looking at these four behemoths, the two Reds fans immediately sat down and were not heard from again.

The incident came to mind recently with the controversy over racist taunts yelled by Red Sox fans at Baltimore Orioles outfielder Adam Jones at Fenway Park. While fans can often enforce basic rules of civility like our four Wrigley friends, there remains a serious problem with unruly fans. In the case of Jones, some fans used the game to vent their racist rage. In response, well-known baseball sports agent, Scott Boras, called for Congress to criminalize racist comments at sporting events.

While I share the anger over the incident, the criminalization of speech is not the answer. It is the ultimate “change up” constitutional pitch. It starts as a response to vile racist speech but ends up curtailing free speech as a whole.

I detest loud, insulting fans. Having grown up near Wrigley, I cannot say that swearing was never heard in that sacred place but it was far less common than other parks. I now live in Washington, D.C., where I rarely took my four kids to stadiums because of the increasingly presence of drunken, profane idiots who ruin the game for other fans, particularly families. They have thrived in the absence of action by team owners who have every right to remove disruptive fans.

However, Boras wants to go further. He demanded, “We need legislative and congressional action so that the awareness of this provides an appropriate penalty for the crime, to give the owners and teams the ability to make conduct from fans criminal so that there is a real benefit to this newfound resolve.” Boras added, “If you run on the field, that trespass gets you jail time and a criminal act. Why shouldn’t conduct of this nature get obviously a greater penalty, because it has worse damages?”

The reason is that “free trespass” is not protected under the first amendment, but free speech is. Even racist speech. If Congress were to criminalize racist speech, it could criminalize a wide array of speech that is deemed insulting or demeaning. That is precisely what is happening in Europe where free speech is being sharply curtailed to combat insulting or intimidating or threatening comments. As a result, everyone from politicians to comedians have been charged under ambiguous speech standards.

The United States remains a relative bastion of free speech in the world despite clear challenges on our college campuses. In R.A.V. v. City of St. Paul in 1992, the Supreme Court considered a St. Paul ordinance criminalizing hate speech after a cross was burned on the front yard of a black family. Associate JusticeAntonin Scalia wrote the opinion that struck down the ordinance as unconstitutional under the First Amendment.

Likewise, in 2011, the Supreme Court considered a case involving the infamous Westboro Baptist Church and its signature protests featuring homophobic slogans at funerals for service members. The Court ruled 8-1 in Snyder v. Phelps that the hate spewing speech celebrating the death of a soldier was still protected under the First Amendment.

The Jones incident should not be unexpected. With 30,000 or so fans, it is hardly surprising to find a handful of virulent racists. Just add enough alcohol and adrenaline and social inhibitions are stripped away to reveal the most ugly aspects of some people. The answer to incidents like the abuse of Adam Jones is not the criminalization of speech but the enforcement of an owner’s right to exclude disruptive fans.

Every year it seems to get worse, but it does not have to happen. As Nez Balelo, Jones’ agent, insisted that “[t]here needs to be a ‘Zero Tolerance’ rule put in place across MLB to punish people that are going to act this way.” The rule would go something like this: act like an adult or leave the park. That means that you do not get to force other people to hear your racist diatribes.

However, it should not just be limited to racist or sexist or anti-Semitic slurs. These parks are a wonderful American tradition. Families should not have to stay at home to protect their kids from some drunk moron who believes that screaming profanities or abuses at a player will finally bring meaning to his sad life.

You also do not get to abuse fans of opposing teams and call it sport. In 2013, the Seahawks hired undercover agents to wear the colors of the 49ers to catch thugs who threaten or abuse other fans. To the team’s credit, they were seeking to bust their own fans to keep the stadium a fun place for all of the fans.

Such private action is not government regulation of speech barred under the First Amendment. Ballparks are private property and fans must adhere to the rules of the game just as do the players. In the same fashion, a theater can throw out obnoxious people who cannot take a break from their cellphones.

In the case of drunken or abusive fans, they are actively trying to disrupt players or the game. Notably, under Rule 901(d), any umpire can eject “any player, coach, manager or substitute for … for unsportsmanlike conduct or language.” The same should be true for fans who engage in unsportsmanlike or obnoxious conduct.

So leave the first amendment and the criminal code out of it. The problem is that obnoxious fans believe that they have license to abuse players and fans because no one is stopping them. Just as umpire Bill Klem once said, “it ain’t nothin’ till I call it.” It is time to call out fouls in the stands and eject those fans who see players and fans as their captive audiences for venting hateful or abusive thoughts.