The Burlington Coat Factory was in such a hurry to distance itself from a former storefront that’s now a highly controversial proposed mosque near Ground Zero that it severely damaged the building when it ripped its old signs off, the property owners charge.

In papers filed in Manhattan Supreme Court, 45 Park Place Partners and 51 Park Place LH LLC say the chain’s “animus and hostility to the inclusion of the mosque in the development plans of the building” resulted in the company “illegally and forcefully” removing its massive signs, destroying the owners’ property and causing them “economic injury.”

The suit seeks a total of $4.1 million in damages.

A rep for Burlington Coat Factory declined comment.

The store had been located inside 45 Park Place until its lease was terminated in 2008.

“Throughout Burlington’s tenancy, and . . . for years before, affixed to the building were two large signs,” one that was 129 square feet, and another that was 32 square feet, the suit says. The signs were bolted and permanently attached to the building, and provided “plaintiffs a valuable asset which increased the value of the premises and provided rental income from advertising on the sign.”

In late 2009, the new owners announced plans to build a community center on the site which would include a mosque, and by last May, the site had become “the focus of intense media attention” and the “subject of controversial protests” because of its “perceived proximity” to Ground Zero, the suit said.

That’s when Burlington apparently decided to distance itself from the project by tearing the signs down, even though it was “without any authority to do so,” and “did not have valid municipal permits to undertake any activities at the building.”

The suit seeks restitution for the physical damage to the building, as well as punitive damages for the company’s “wanton and willful trespass.”