(CNN) -- A Pennsylvania day care center is pursuing a federal civil rights lawsuit against a swim club that canceled the center's membership, the day care director and attorney told CNN's "American Morning" on Wednesday.

Some kids from the Creative Steps Day Care Center say white members of a swim club made racial remarks.

"Specifically, the lawsuit will be based on violations of Title II of the Civil Rights Act of 1964 which relates to the denial of access and use and advantages of a public accommodation," attorney Carolyn Wright said.

"The humiliation, embarrassment, and the scarring and suffering of the children -- that would be covered by our claim for compensatory damages."

Wright said the center -- attended by many minority children -- is also seeking "punitive damages to punish the club for the harm that they've done to the children."

The dispute started after the Valley Swim Club, in Huntingdon Valley, Pennsylvania, revoked the swimming privileges of about 65 children from the largely minority Creative Steps day care center after one visit on June 29.

Some black and Hispanic children said white club members made racist comments to them during that visit, asking why "black children were there" and raising concerns that "they might steal from us."

The day care center had originally contracted to use the pool during the summer, but the club canceled the agreement and returned the day care center's $1,950 check without explanation.

Don't Miss

The club canceled contracts with two other day care centers because of safety and crowding, swim club director John Duesler said.

Those two other centers have not protested the club's actions.

The issue was exacerbated when Duesler told two Philadelphia television stations the children had changed "the complexion" and "atmosphere" of the club. The comment brought protesters outside the facility.

Duesler later said that safety and crowding -- not racism -- prompted the cancellation.

As the controversy grew and reached national news shows, the swim club asked the Creative Steps day care to return. It refused and said it would pursue a lawsuit.

Creative Steps Director Alethea Wright told CNN that the club did not reach out directly to her facility to invite them back to the pool.

"My attorneys have not received any written invitation for us to come back," Wright said Wednesday. "The only thing that I've heard has been third party via the media."

She said she offered to come to a resolution but she said Duesler told her following a meeting of the club's board, "'Let the chips fall where they may.'"

The swim club has been subpoenaed by Pennsylvania's Human Relations Commission, which has begun a fact-finding investigation, said the wife of the club's director, Bernice Duesler.

She told CNN on Tuesday night that the club "could not handle a lawsuit."

"We don't have the money. We're just a small club. We're volunteers," she said. "I hope we can teach our children a lesson -- that you should admit errors. We should have done things differently. And if there are differences, we can overcome them."