“Because they’re the only ones that don’t have to look at Harbor Towers.”

True enough. The Harbor Towers are ugly, products of the same architectural insanity that gave us Boston City Hall, the State Service Center, and, adjacent to the Harbor Towers, the garage. The towers and the garage were both designed by I.M. Pei, a saint to architects — proof, I guess, that even saints can sin. The hulking, impersonal garage is particularly malign: It tells you much about our past attitude towards the water, and the way we blithely polluted it, that the best use for such frontage was a place to store idle automobiles. So one would think that everyone with an interest in the renaissance of Boston’s waterfront would welcome proposals by the garage’s owner, Don Chiofaro, to tear down the building and replace it with something better, something beautiful.

And pretty much everyone does seem welcoming, with one ironic exception. The residents of the Towers — some, but by no means all — have girded for battle. On July 21, the Towers’ trustees greeted Chiofaro’s plans with a scathing 12-page letter. The critique struck wearisomely familiar themes: too high, too dense, too much. “We support redevelopment of the Harbor Garage site,” one of the trustees insists to me, but when I ask what, if anything, would satisfy them, I get this response: “Let them propose something and we’ll see.” It’s an old game: Make developers negotiate against themselves.

The public has a deep and abiding interest in what becomes of the entire waterfront.

Why the opposition? Perhaps, once the garage is done, they fear they’re next. Perhaps it’s mere NIMBYism. Or perhaps it’s because the towers are dependent upon the garage: It houses not only resident parking (a deal that’s due to expire in 2022) but also the buildings’ HVAC and mechanical systems.

All of this confrontation has created the widespread perception that the battle over the garage is basically a fight between the towers residents and Chiofaro. That’s a shame. It isn’t, and it shouldn’t be. This is everybody’s battle.

Cut off from the city by the elevated Central Artery, Boston Harbor was once a squalid and inaccessible mess. Today, thanks to the Big Dig, the Deer Island sewage treatment plant, and efforts by innumerable individuals and organizations, the city is reclaiming the water. A crucial player in that has been Vivien Li, head of the Boston Harbor Association and a driving force behind the creation of the HarborWalk, a path of waterside public access that extends for 39 miles. She also sits on a key harbor review commission, one of the many entities that will be reviewing Chiofaro’s proposal.

Chiofaro’s plan has breathtaking beauty. It envisions two towers, one terra-cotta, the second glass. Uses would be varied, including offices, residences, retail, and a hotel. The two towers would be connected by public space with a retractable roof. Views from the street to the water would be plentiful, and the city skyline would be strikingly improved.

A good start, in other words, but there’s still a long way to go. The project’s mass may, indeed, be excessive. Li points out to numerous concerns, including shadowing, wind and congestion. Creative thought about mitigation that might lessen their impacts, she says.

More broadly, she argues, buildings can’t be viewed in isolation, but rather have to be considered as part of the fabric of the area. For example, can the plans for the Harbor Garage be a spur for rethinking the space in front of the New England Aquarium?

Still, she and many others — such as the activist group BostonNow — frame these issues in the context not of how do we stop it but, rather, how do we make it work? Chiofaro’s plans have provoked much discussion, and for good reason: The public has a deep and abiding interest in what becomes of the entire waterfront. Harbor Towers has a loud voice, no doubt. But it’s everyone else’s that deserves a listen.