Members of the North Carolina-based Loyal White Knights of the Ku Klux Klan -- some wearing Klan robes and carrying Confederate flags -- arrived in midday to protest the city's plan to remove a statue of Confederate Gen. Robert E. Lee from a park, one of several steps the city is taking to reduce its number of Confederate monuments.

Some Klansmen had said they would carry loaded firearms -- allowable under Virginia's open-carry law, CNN reported.

They were confronted by shouting counter protesters as tensions ratcheted up and a large group of law enforcement officers, including members of the state police, set up police lines.

Police escorted the Klansmen through the shouting crowds to reach their designated spot for a demonstration: Justice Park, which until recently had been named after Confederate hero Gen. Stonewall Jackson.

About 1,000 counter protesters were there, said city spokeswoman Miriam Dickler.

After about 40 minutes, police helped the KKK members push through the crowd to reach their vehicles, Dickler said. Some protesters tried to block the Klan vehicles and police then declared the gathering an unlawful assembly, Dickler said.

As police began to move back toward Justice Park, a large crowd followed, Dickler said. There were a number of incidents, including the use of pepper spray by the crowd, Dickler said. When the crowd didn't follow orders to disperse, state police released three canisters of tear gas and the crowd left, Dickler said.

She said three people were taken to the hospital -- two for heat-related issues and one for alcohol usage.

Twenty-three people were arrested. It was not clear what they were charged with or whether those arrested were Klan supporters, counter protesters or some from each group.

The Loyal White Knights of the KKK said the rally was called to support Southern heritage, CNN affiliate WVIR reported.

The city and the University of Virginia had urged people not to go to the Klan rally, even if they opposed the Klan, but to attend alternative events, such as a unity concert at a school. The local chapter of the NAACP also organized a counter-rally.

"Throughout the day, we urged folks not to take the bait -- to deny the KKK the confrontation and celebrity they desire," Mayor Mike Signer said on his Facebook page. "Thousands of people followed that advice, attending a wonderful set of alternative events. ..."

Charlottesville, home of the University of Virginia, has, like many cities and towns, grappled with what to do with its Confederate monuments.

"Charlottesville has kind of been put on the map recently," the mayor said earlier in the day. "We want to change the narrative by telling the true story of race through public spaces. That has made us a target for groups that hate that change and want to stay in the past, but we will not be intimidated."

The city council voted in April to remove the Lee statue and sell it by taking bids, CNN affiliate WVIR reported. Lee Park, where that statue stands, was renamed Emancipation Park.

Those efforts were praised by the local chapter of the NAACP, which said in a statement: "The NAACP has committed to removing and advocating the removal of Confederate symbols, including statues and from flags in cities all around the nation. In Charlottesville, they continue to support the call to remove the statue of Robert E. Lee."

This is not the first protest over the removal of the Lee statue.

In mid-May, a group carrying torches and led by prominent white nationalist Richard Spencer held a demonstration around the statue to protest its removal. The gathering was swiftly condemned by city leaders, who said it evoked images of the Ku Klux Klan.