With its quiet residential streets, grand apartment buildings, cool air sweeping over the Hudson River, and waterfront esplanades lined with cherry blossoms, New York’s Upper West Side, where I was born and raised, has never looked better. It’s easy to forget that, starting in the 1970s, the neighborhood descended into chaos as a coalition of politically connected developers, nonprofits, labor unions, and government agencies did its utmost to turn the area into a dispensary for social services. Under the cover of compassionate rhetoric, the groups used public funds to convert the Upper West Side’s private residential buildings into welfare hotels, homeless shelters, halfway houses, and methadone clinics, inundating the neighborhood with crime, homelessness, and drug abuse.

The Upper West Side may be known for its affection for leftist causes and Great Society paternalism, but its citizens, seeing the destructive consequences of the policies that resulted, valiantly battled this government-sponsored assault on their community. Their fight began in earnest with the cooperative revolution, which transformed rental tenants into owners with the power to reclaim their broken neighborhoods. These residents—through their co-op boards, self-started nonprofits, and free associations, and with some help from city hall, after Rudy Giuliani’s election—have worked tirelessly to restore the beauty of the Upper West Side. Block by block, tree by tree, they have turned this neighborhood from a forsaken wreck into one of the most tranquil and desirable in the city.

The battle has continued to rage, though. On one side is gentrification, propelled by the new ownership society; on the other is institutionalization, pushed by an agenda that depends on devaluing the neighborhood’s housing stock, the better to convert it into social-services facilities. The balance has tilted toward gentrification, but several recent reversals show that the gains can’t be taken for granted.

Thanks to its rocky terrain, the Upper West Side was first developed a generation later than its Upper East Side counterpart. Starting in the 1860s, city planner Andrew Haswell Green commissioned two new parks for the area from Frederick Law Olmsted and Calvert Vaux, who had already designed the adjacent Central Park. Olmsted and Vaux designed Riverside Park and Morningside Park in a way that departed from the street grid, followed the curves of the natural landscape, and incorporated monuments, footpaths, and promenades. At the same time, Broadway was extended north through the Upper West Side as a grand boulevard roughly following the old Bloomingdale Road, cutting on a bias from 59th Street. With its asphalt rather than cobblestone surface, the street became a popular destination for recreational cyclists during the city’s first biking craze, in the 1890s.

The neighborhood, liberally adorned with green space, became arguably the most beautifully laid out in the city. Following the initial wave of single-family mansions and townhouses, developers commissioned the best architects of the Gilded Age to design apartment buildings along Riverside Drive, Broadway, and Central Park West. Highlights of the period include what are still regarded as the area’s most beautiful buildings: the Dakota, crafted in a German Renaissance style by Henry J. Hardenbergh in 1880; the Ansonia Hotel, designed in the beaux-arts style by Paul E. M. Duboy in 1899; the Apthorp, a full-block Renaissance Revival palazzo designed by Clinton and Russell in 1906; and the Paterno and the Coliseum, two gracefully curving buildings at Riverside Drive and 116th Street designed by Schwartz and Gross in 1908 and 1910. These and the other grand buildings that went up before World War I were even more deluxe than many of the “prewar” (that is, pre–World War II) apartments that, built a generation later, now define the Upper East Side.

Yet despite the presence of wealthy residents like William Randolph Hearst, who occupied several floors of an apartment building on 86th Street and Riverside Drive, the area never caught on as Manhattan’s most desirable location. By the 1930s, landlords had subdivided many of the neighborhood’s most spacious apartments into more affordable rentals (the concept of apartment ownership remained far off). Townhouse and brownstone owners broke up their single-family homes and took in lodgers. My building, which Schwartz and Gross designed in 1909, went from three apartments per floor to eight, with maid’s rooms converted into separate residences and the servants’ passage converted into a public hallway. Later divisions turned some floors into nine- and ten-unit layouts, with a communal kitchen down the hall.

Meanwhile, Olmsted’s parks were taking a beating, as Hoovervilles went up at 72nd Street and Riverside Park and on the Great Lawn of Central Park during the Great Depression. It didn’t help that the New York Central rail line, with meatpacking trains burning oil and coal, ran uncovered along the Hudson River for the length of Riverside Park, depositing a layer of ash over the area. In the 1930s, Robert Moses, who looked out on the tracks from his Riverside Drive apartment, commissioned what would be the last great public works improvement for the neighborhood, dramatically enhancing the beauty of Olmsted’s original plan. For twice the cost of the Hoover Dam—$100 million—he covered the tracks from 72nd to 120th Streets, extended Riverside Park farther out into the Hudson, and wrapped the new Henry Hudson Parkway through the addition in a way that opened the waterfront to foot traffic.

The improvements could do little, however, to maintain a neighborhood that was becoming a hostage to rent control. During the 1940s, the federal government froze prices, including rents, nationwide. Most of those price controls eventually expired, but in 1950, New York State decided to turn a temporary wartime act into a permanent provision by capping the rent on 2.5 million units—85 percent of them in New York City—that had been built prior to 1947. Some years later, the state enacted rent stabilization, which regulated how quickly landlords could increase rents, for units built between 1947 and 1974.

These laws were born from a fear of the free market, but they wound up showing how much more destructive government-manipulated pricing was. For one thing, they artificially limited the size of the market for uncontrolled rental units and thus raised the rental rates in new construction. Average, honest renters in rent-controlled apartments found themselves unable to afford to move to unregulated ones, even if their buildings were collapsing or if their housing needs were shifting—say, because of changing family size. At the same time, the wealthy could maintain multiple residences while keeping their sprawling and underused rent-controlled apartments off the market. The corrupt learned to file phony complaints with the Department of Buildings and to launch bogus “diminution of services” lawsuits, tying up rate increases in litigation (the practice remains commonplace today).

A still more destructive effect was that the state-sanctioned rent increases were too small to let owners operate and maintain their aging buildings. In the 1970s, small landlords all over the city were squeezed further as the price of fuel oil rose; many couldn’t pay their property taxes, and their buildings became unprofitable to own and worthless to sell. “In New York City,” wrote economist Thomas Sowell, “many buildings have been abandoned after their owners found it impossible to collect enough rent to cover the costs of the services that they are required by law to provide, such as heat and hot water.” Landlords defaulted or walked away from their buildings, rending the neighborhood.

Among the many consequences of this real-estate collapse was a major government intrusion into the city’s housing market. As housing expert William Tucker wrote in these pages in 1990, the city government seized thousands of buildings over back taxes in the 1970s and 1980s. Rather than quickly returning its acquisitions to the private sector, as most American cities did, New York held on to them. An agency today called the Department of Housing Preservation and Development (HPD) set about managing seized properties and also creating vast new developments underwritten by government subsidies. (This agency is separate from the New York City Housing Authority, or NYCHA, which operates low-income subsidized housing.) “By 1984, HPD held title to 8,000 buildings containing 108,000 apartments—62,000 occupied and 46,000 vacant,” Tucker calculated. By the time he was writing, the HPD and NYCHA together controlled a staggering 16 percent of the rental market. The city had become both New York’s largest landlord and its largest developer.

One feature of the HPD’s reign during the 1970s was a program in which housing courts would appoint administrators for “buildings whose owners had proved incapable of handling them,” Tucker wrote. Such an administrator “was not obligated to pay mortgage installments, insurance premiums, or property taxes—items that normally account for 60 to 80 percent of a landlord’s expenses”; those obligations remained with the building’s owner, while the administrator got the rent money. It was an exceedingly attractive arrangement for the administrator, and “appointments quickly fell to tenant activists, community-organization leaders, and friends of housing court Judges.”

The ready supply of housing stock also enabled a rising political class to funnel money from Great Society agencies, such as the federal Department of Housing and Urban Development, through their local political mills to “revitalize” properties as government housing. Historic buildings were leveled, too, to make way for superblocks of new subsidized developments. In contrast with the uplifting designs of the neighborhood’s original Gilded Age construction, the new buildings that went in along upper Columbus Avenue in the 1960s, many of them under the Mitchell-Lama subsidized housing program, were oppressively ugly, their Brutalist architecture signaling the government’s new dominion over the neighborhood.

Private developers tried to step in with their own capital to improve the neighborhood, but local politicians slammed the door on them—and none more rudely than Ruth Messinger, an Upper West Sider who served as city councilwoman from 1978 to 1989 and then as Manhattan’s borough president before losing her mayoral bid to Rudolph Giuliani in 1997. A member of the Democratic Socialists of America, she launched her political career by opposing private developers who wanted to convert the area’s prewar buildings into condominiums. She also advocated extending rent control from the neighborhood’s tenants to the businesses that leased ground-floor space, a move designed to starve building owners of their last remaining source of revenue. (The issue hasn’t gone away: just this year, local politicians have proposed legislation to limit the size of neighborhood stores, forcing out larger, higher-paying tenants.)

What saved the Upper West Side’s buildings—aging, stocked with rent-regulated tenants, and apparently destined to fail as private assets—was the advent of the co-op. Under cooperative ownership, a building’s residents are shareholders in a corporation that owns the entire building, often with a percentage of the building retained by the original landlord, known as the “sponsor.” The corporation then grants the residents the use of their apartments through a proprietary lease. Unlike most multiple-dwelling condominiums, in which residents own their apartments outright, cooperative corporations can maintain underlying mortgages on their buildings, much as a single owner would have a mortgage on a house. It is for this reason that the owners of rental buildings are able to convert their properties into cooperatives: the corporation can take on the building’s existing mortgage. That’s also why cooperative boards must maintain high financial standards for incoming shareholders. A default by even one resident can trigger a default on the entire building.

The cooperative ownership of multiple-dwelling buildings by residents had been around since before the twentieth century, but it wasn’t until the 1980s that the practice really took off on the Upper West Side. A number of factors were responsible, mainly a real-estate boom fueled by the growth of Wall Street and a new state law that made the process easier by lowering the minimum percentage of a building that a landlord had to sell—to 15 percent on non-eviction-plan conversions. Co-ops allowed private landlords to remove apartments from the dominion of rent control while also sharing the costs of building maintenance with a new class of owners.

My building underwent such a conversion in 1987, offering an insider rate for current renters to buy while letting those who opted out remain in their existing apartments under what is known as sponsor ownership. As with most buildings that underwent such a conversion, the result was transformational. Rising apartment prices brought in buyers with capital and willing to invest in the building’s revitalization. Through assessments and refinancing, my building could now afford to put in new windows, repoint its brick-and-limestone facade, and replace the missing pieces of its elaborate parapet. A new laundry room went in. The upper hallways were retiled and refurbished. The roof was replaced. All these improvements were agreed upon by a board of shareholder volunteers.

The cooperative spirit didn’t end at the property line. Co-op shareholders all over the Upper West Side led the charge to take ownership of the surrounding community, and they brought civic-minded renters and condo owners with them. In 1977, for example, a group of residents began planting flowers in a vacant lot at Broadway and 96th Street. When new condominium apartments were built there in 1981, the gardeners moved their operation to a dirt median on the Serpentine, a walkway in Riverside Park over Robert Moses’s train tunnel, and formed a nonprofit group called the Garden People, whose flower garden is one of the finest in the city and has served as a backdrop in movies. Also in the 1980s, a group of neighborhood mothers came together to reclaim an aging playground in Riverside Park from rats and vagrants. They raised funds to turn the space into the Hippo Playground—a whimsical play area festooned with sculptures of hippos, turtles, and frogs that spray water in the summer months.

Around the same time, my neighbor Amanda Larrick decided to do something about Joan of Arc Park, a little archipelago of parkland at Riverside Drive between 91st and 95th Streets. “There was not a blade of grass anywhere to be seen,” she remembers. “It was in such terrible, terrible shape. A dust bowl. We had these lovely crab-apple trees, and people would just break off the branches. One day, I said, ‘Let’s stay out there,’ and we saw these people, and we asked them to stop, and they did.” Larrick and her neighbors became volunteer park tenders, and they continue to meet every Saturday, from April through mid-December. From nine in the morning until one in the afternoon, they sweep ginkgo fruit out of cobblestone seams and rotting leaves out of gutters. They gather fallen twigs and spread wood chips across pathways. Armed with a pruning permit, they take down loose branches with a pole saw. And they plant: holly trees, crab-apple trees, quanson cherries, oakleaf hydrangea, wild azaleas, quince bushes, and periwinkles. Last fall alone, they put in 300 new flower bulbs supplied by the Riverside Park Fund—the great resident-led nonprofit that has revitalized Riverside Park. Larrick says that she hasn’t missed a Saturday since she started the project. “I don’t even have to call people any more,” she says. “They know we are there.”

Unfortunately, the cooperative revolution came to a halt in the late 1980s and early 1990s, a casualty of economic recession. And the same recession opened the door to a new invasion of the Upper West Side, one that continues to this day. Nonprofits and their allies in the social-services sector spied an opportunity in the newly depressed housing market, which gave them the purchasing power to rent and buy properties where tenants still lived. On the Upper West Side, they focused on SROs—buildings with “single-room occupancy” apartments, tiny rooms whose occupants used communal bathrooms and kitchens. The idea was to import and house people with both mental-health problems and drug dependency, a dual diagnosis known in the industry as MICA (for “mentally ill, chemically addicted”). Placing a MICA population in what is known as “supportive housing” secured the highest possible government funding for the social-services agencies. The agencies implanted these profoundly troubled new residents alongside the tenants who remained in the SROs—a process that frequently drove the tenants out.

Jeffrey Goldberg documented the situation in “The Decline and Fall of the Upper West Side: How the Poverty Industry Is Ripping Apart a Great New York Neighborhood,” a groundbreaking article for New York in 1994. “Small business is no longer the dominant industry of the Upper West Side. Homelessness is,” he wrote, echoing much that had been said in the pages of City Journal. Even as the city was going broke, the “explosive growth of the social service sector” meant that the neighborhood had 80 city, state, and private social-services facilities, with five new ones opening each year. Not only did the influx of undesirable new residents drive rent-paying tenants out of their apartments; the worst-off of those tenants were forced into the streets and thus into the hands of the homelessness industry, which housed them at four times the cost in the same SROs that they had been pushed out of in the first place.

By 1994, Lisa Lehr, the chairwoman of the West 90s West 100s Neighborhood Coalition, was lamenting how the area had become an “open-air asylum.” One local crack-addicted panhandler, Larry Hogue, personified the problem and achieved national infamy. Known as the Wild Man of 96th Street, he rotated in and out of mental-health facilities—30 times or more—for seven years, always returning to the Upper West Side to convert his government checks into weeklong drug binges in which he would set fire to cars, masturbate in public, and throw objects at traffic. He once pushed a girl in front of an oncoming truck.

The rising disorder on the Upper West Side—Hogue and other addicts, crack vials on the streets, homelessness run amok—only helped the social-services industry buy up housing stock more easily, as residents fled the neighborhood. The obvious public presence of a mentally ill population also allowed politicians to claim that the need for social services in the area was growing. The truth, of course, was that the rise of the social-services agencies was what had introduced that population in the first place.

On January 1, 1994, Rudolph Giuliani took office as New York City’s first Republican mayor since 1965. Carrying out his campaign promises to take on “the street tax paid to drunks and panhandlers, . . . the squeegee men shaking down the motorist waiting at a light, . . . the trash storms, the swirling mass of garbage left by peddlers and panhandlers, and open-air drug bazaars on unclean streets,” Giuliani brought new policing strategies to New York—among them the Broken Windows theory of order maintenance, which holds that crime is less likely in well-maintained neighborhoods. The policies worked wonders on the Upper West Side, partly because the new mayor was carrying through a quality-of-life campaign that the cooperatives had already started. After all, the co-op revolution had literally fixed the neighborhood’s broken windows, along with its flower beds, monuments, and playgrounds.

With a partner in the city’s highest office, the neighborhood rebounded as never before. The Lincoln Center area, around 66th Street, saw the arrival of luxury high-rises, big-box stores, and a movie theater. The local political machine, moving even further left as a check on Giuliani’s alleged excesses, railed against the new development. “They want to put in the largest bookstore in the world and the largest record store in the world—and a ten-plex movie theater,” lamented Elizabeth Starkey, the chair of Community Board 7, which encompasses the Upper West Side. “Do you realize what a ten-plex movie theater does to a neighborhood?” Of course, the answer was clear to everyone: these improvements boosted the area’s quality of life and raised apartment prices considerably.

They also pushed gentrification farther uptown, into the West 90s—which is where the battle for the Upper West Side is currently being fought, particularly in blocks with high concentrations of SROs. Aaron Biller, president of the resident organization Neighborhood in the Nineties, describes the process as an ongoing game of “Whac-A-Mole” in which the social-services industry pops up in one building after another.

Take St. Louis Hall, a six-story residence on 94th Street, just steps from Riverside Park. Over the years, as the neighborhood gentrified, what was once a notoriously run-down SRO began to attract better tenants. But then the Lantern Organization, a nonprofit housing developer, and its for-profit wing, the Lantern Management Group, bought the building and began converting it into a facility for a MICA population. Lantern originally agreed to move the existing tenants into a section of the building separate from the one where the troubled new clients would live.

But now, Lantern is attempting to push the existing tenants out by threatening to house them side by side with the MICA population. Lantern’s motivation is clear: for each “special-needs” tenant whom it can squeeze in, investors can earn more than $3,000 a month from government agencies paying to house them. Protected by rent stabilization, existing residents, who might pay only $500 a month, stand in the way of maximum profits. “To scare the Hispanic tenants, they had someone yelling ‘immigration,’ ” Biller tells me of Lantern’s tactics. “They distributed flyers saying they are bringing in a population with AIDS. They are driving people out and have a huge economic incentive to do it.” Neighborhood in the Nineties has litigated against Lantern’s conversion plans for St. Louis Hall and succeeded in reducing its size somewhat.

In 2008, after conducting an investigation into Lantern, the local CBS station found that the organization “took millions of dollars from the city to provide clean, safe and affordable housing for the mentally ill, recovering drug addicts and others in need” but instead put them in “deplorable conditions.” At the St. Louis, CBS reported “deteriorating conditions under Lantern’s ownership,” including “rats, mice, roaches, bedbugs, and . . . dangerously toxic black mold.” When CBS tracked down Lantern’s president, T. Eric Galloway, at his 6,000-square-foot mansion in upstate New York, he refused to comment.

A close relationship exists between the supportive-housing industry and the government agencies that fund it. Before becoming the executive director of Lantern, Jessica Katz worked at the Department of Housing Preservation and Development, where she was “responsible for an annual Supportive Housing pipeline worth over $100 million,” according to Lantern’s website. The department lent over $15 million of that money to Lantern interest-free to pay for its conversion of St. Louis Hall. As Biller puts it: “Jessica Katz worked for HPD, set up all the financing, then immediately went to Lantern.” Soon after I reported the situation in the Daily News, Katz left Lantern.

Robert Atkins is a professional musician who has lived at the St. Louis for five years. “If this were a building with a predominantly white population, middle-class, they wouldn’t even try to get away with this,” he says. “This whole affordable-housing thing is a hoax. It’s not affordable to the taxpayer. It’s not affordable to the poor. The only people who are making out on it are doing so at the taxpayer’s expense.”

One of the latest skirmishes in the war involves a creative way that SRO landlords have found to keep their buildings afloat: turning them into European-style budget hotels. The regular tenants who live among the hotel rooms often applaud the change, since it means new services and additional security and staff. Local merchants also welcome the tourist trade.

But the neighborhood’s political leadership thought otherwise. In 2006, Councilwoman Gale Brewer, Ruth Messinger’s onetime aide and now heir apparent, teamed up with state legislators Richard Gottfried, Linda Rosenthal, and Liz Krueger to outlaw the new hotels. The result, of course, was to leave the SROs at the mercy of the social-services network. Late in 2010, I watched as one building, the Alexander Hotel on 94th Street, signed a contract with Samaritan Village, a Queens-based substance-abuse and mental-health center that began converting parts of the building into a 200-bed homeless facility—a troubling development for the building’s permanent residents.

After I broke the story for the Daily News, Neighborhood in the Nineties led a grassroots campaign against the project. Neighbors flocked to community-board meetings, which got sympathetic TV news coverage. At one meeting, Biller called out Brewer to thunderous applause—a sign that radical politics no longer enjoys unquestioned dominion in this liberal neighborhood. Soon after, Brewer arranged a press event on the steps of the Alexander to denounce the conversion, even though a Department of Homeless Services spokeswoman, according to the website Gothamist, insists that the agency had “closely communicated with Council Member Brewer and Community Board 7 regarding the proposal on several occasions.”

A “fair share” law in the city charter requires social-services facilities to be evenly distributed through all neighborhoods, but West 94th and 95th Streets alone have seen half a dozen similar institutions proposed in recent years, from homeless shelters to drug-treatment centers to halfway houses. In 2008, Neighborhood in the Nineties released a study of data from the Supportive Housing Network confirming what the residents of the Upper West Side had long suspected: a full 21 percent of the supportive-housing units in Manhattan—1,978 units—were located on the Upper West Side. On the Upper East Side, by comparison, there were 93 units.

The Upper West Side is known for its social compassion, but that compassion has long been abused by developers and politicians who profit from the failure of the neighborhood. More and more, my neighbors are realizing what’s going on. For the moment, our neighborhood remains beautiful and vital—but the battle for the Upper West Side is far from over.

Research for this article was supported by the Brunie Fund for New York Journalism.

James Panero is the managing editor of The New Criterion. His Twitter handle is jamespanero.