The Gadsden flag is causing quite an uproar in New Rochelle, N.Y. (Photo: CBS 2)

NEW ROCHELLE, N.Y. (CBSNewYork) — A Westchester County veterans’ group is taking the city of New Rochelle to court.

The lawsuit is all over a flag that was being flown over an empty building, CBS 2’s Janelle Burrell reported Monday.

It’s a flag steeped in American history.

“George Washington’s army, the Continental Army actually fought under that flag,” said Ron Tocci of the United Veterans Memorial & Patriotic Association.

But in New Rochelle, it’s brewing debate, because of its association with the Tea Party movement.

Last month, New Rochelle officials had the “Don’t Tread on Me” flag removed from the city’s vacant armory building, because they feared that the flag may be seen as political.

Now, members of a local veterans group are suing the city.

“It has no association with any one particular group. No one owns that flag but the United States of America and that’s the freedom that we want to prevail,” said Peter Parente, president of the United Veterans Memorial & Patriotic Association.

They argue that the so-called Gadsden flag has historical significance.

“It flies at military bases, so in no way shape or form do we see this flag to be offensive. It shouldn’t be offensive to anybody,” Tocci said.

City leaders disagree.

“The bottom line is there was no permission asked for or granted to fly this flag on city property, and therefore it was removed,” New Rochelle City Manager Chuck Strome said.

Strome said some on the city council perceive the flag to be divisive.

“Some members of the public and city council have felt that flag was adopted by the Tea Party and took offense to having a political statement on public property,” Strome said.

The building hasn’t been used for decades, and city officials said even though it was once used by the military, they reserve the authority to decide what flies above it.

The veterans claim it’s a violation of their rights.

“The ultimate word will come from the courts,” Tocci said.

Most residents CBS 2’s Burrell spoke with said they support the vets and their fight.

“What the council did and the mayor, I don’t think they deserve to hold public office for voting against the vets,” resident Antoinette Pulice said.

“It’s historical, and it’s their right to fly it,” John Banks added.

But for now, the red white and blue stars and stripes is the only flag that will fly.

Some members of the city council who voted against the flag said they have been receiving racially charged hate mail since the decision. The veterans’ group said it condemns any offensive and hate-filled backlash.