There’s an alarm on the door leading to the roof of the Hotel Chelsea, but the sound it makes is so quiet and pitched so high, somewhere between a dog whistle and an old-fashioned kettle on the boil, that it took me a while to realise we had tripped it. “There are cameras everywhere,” Sherill Tippins warned, but if anyone on the staff had noticed us, they didn’t object.

The rooftop garden had been dismantled, plant pots emptied and shoved into corners. Most of the windows were dark, the apartments vacant. But it was a thrill to be up there, to imagine one more party and to invite the denizens, living and dead: Arthur C. Clarke and Bob Dylan. Arthur Miller and Brendan Behan. Mark Twain and Andy Warhol. Milos Forman, Janis Joplin, Dylan Thomas and Leonard Cohen.

Inside, the corridors had been stripped and whitewashed. Occupied flats were covered in clear plastic sheeting, with red zips, to protect the people living there from asbestos dust. The empty ones were boarded up. But on the tenth floor we found a door ajar, leading into a dingy space with a tiny sink in the corner and a shared bathroom across the hall. “Patti Smith stayed in a room this size, on this floor. It could be hers,” Tippins offered. “She doesn’t remember which one.”

As we descended the spiral staircase, Tippins reeled off the list of famous residents, floor by floor. “They’ve taken the room numbers off. We’re losing the sense of who lived where. The history is slipping away, which is scary,” she said. Her remarkably thorough and engaging book, Inside The Dream Palace, presents the building as an egalitarian space, creatively fertile by design. Conceived as a “Home Club Association” by architect Philip Hubert in 1883, it has nurtured writers and artists ever since.

The story of the Hotel Chelsea has been told many times in print, in song and on film, but it was a revelation to me to find the place a building site, closed to everyone but the last eighty or so people holding on to their rent-stabilised apartments. It will be an expensive boutique hotel soon enough – there’s no stopping that – but their refusal to let one of the last remnants of bohemian New York be swallowed whole by the luxury economy is an inspiring tale, in this city that has been transformed by, and given over to, international capital.

Tony Notarberberino in his bedroom

When I returned to the Chelsea a few days later, Tony Notarberardino was shooting a short film in the hotel lobby. He wouldn’t say much about it, except that it would be “a little Lynch-ian, Kubrick-esque thing,” and that the deserted corridors remind him of The Shining.

Notarberardino arrived in New York from Melbourne in 1994 and went directly to the Chelsea because a friend was staying there. After one night on the couch, he walked into owner Stanley Bard’s office and asked if he had anything better. Bard assessed him with a look, and after establishing that he could front a month’s rent, took him to a suite on the sixth floor that had recently been vacated by fellow Australian Vali Myers.

“Stanley turned around and said ‘you’ll probably never leave here’ and I thought ‘yeah right,’” Notarberardino told me. “But he was right. I was wrong.” His flat has a working 19th Century fireplace, a man-sized Buddha statue in an alcove in the living room and antique picture frames on almost every inch of wall space. The multi-coloured bedroom ceiling was painted by Myers. “Where am I going to find something like this? It’s impossible. No money would be worth it.”

Bard is a hero to the hotel’s long term residents. His habit of accepting artworks in lieu of rent was one reason the Chelsea attracted so many artists – and his $4.5 million apartment overlooking Central Park suggests he did alright out of it. Although he drew the line at Grateful Dead roadies and live lions in rehearsals for Aida, he tolerated almost anything else. When renowned folk collector and film-maker Harry Smith sneaked out of the hotel, owing tens of thousands of dollars in rent, all Bard wanted to know was how to get him back.

“The Chelsea in the Sixties seemed to combine two atmospheres: a scary and optimistic chaos which predicted the hip future, and at the same time the feel of a massive, old-fashioned, sheltering family,” wrote Arthur Miller, who moved there after his marriage to Marilyn Monroe collapsed. “It was thrilling to know that Virgil Thomson was writing his nasty music reviews on the top floor, and that those canvases hanging over the lobby were by Larry Rivers, no doubt as rent, and that the hollow-cheeked girl on the elevator was Viva and the hollow-eyed man with her was Warhol and that scent you caught was marijuana.”

Judith Childs moved in to the Chelsea in 1966 and is still living there.

Miller eventually found the tumult too much and moved out. Warhol’s film Chelsea Girls applied a lacquer of glamour to the hotel’s reputation. Dylan wrote that after the movie was released “it was all over for the Chelsea Hotel. You might as well have burned it down,” but for as long as Bard remained in charge, the Chelsea continued to be a haven for artists, even as bullet-proof glass was erected at the front desk and the lobby filled with drug-dealers and prostitutes.

In the late nineties, Notarberardino would often come home late to find his neighbour, Dee Dee Ramone, crashed out in the hallway. His soon to be published book, Chelsea Hotel Portraits, documents this era, when the few remaining luminaries from the golden age co-existed with punks and junkies looking for the ghosts of Sid Vicious and his girlfriend Nancy Spungen, who was stabbed to death in a bathroom on the first floor. “They don’t need to change one door handle. That’s my opinion. The place is perfect,” he told me. “But I’m not here to make money.”

Brian Bothwell and Meli Pennington arrived the same year he did, intending to stay for a month, after finding New York landlords unwilling to take a chance on a freelance video editor and a make-up artist. Bard took them up to a one bedroom on the second floor that used to belong to Grace Jones. The outgoing tenants, a British band called the Starlings, were drinking breakfast as they digested the news that they had been dropped by their record label.

Twenty years later, the couple are still there, in an apartment that looks like it was last renovated when Dylan was writing Blonde on Blonde down the hallway. “We were Stanley’s naive children,” Pennington said. “Living in a fantasy art colony,” Bothwell added. “And we loved it. That’s why we got so pissed off because Joseph Chetrit burst the bubble.”

In 2005 Bard’s partners ousted him from the board of directors. Two years later, they sacked him as manager and put the hotel up for sale – just as Wall Street’s mortgage securities bubble was about to burst. While they waited for a buyer, they filled the rooms with whoever would pay and evicted the most vulnerable tenants.

A corridor on the fifth floor. Construction is paused, for now.

In 2011, developer Joseph Chetrit bought the hotel for $81 million and set about ridding himself of unwanted guests, namely long term tenants with protected leases paying below market rate. Those without good lawyers had already been forced out. The rest were encouraged to leave. There were floods, gas leaks and inexplicable power cuts. “They set up an atmosphere of intimidation and harassment and at a certain point we decided to dig in and fight,” Pennington said.

Judith Childs, who has lived at the Chelsea since 1966, described the Chetrit Group as “barbarians and gangsters: arrogant, ignorant, futile, greedy, dishonest…” The day Bard was fired, she removed her late husband Bernard’s paintings from the walls of the hotel. Other less cautious tenants found their artworks confiscated. In the lobby, a resident has affixed a sticker reading “George Chemeche – White Christmas in Pennsylvania, 2006” to the dirty white wall – an arch conceptual joke where a painting by Rivers used to hang.

“Anything you could read in the slumlord handbook we had visited upon us: no heating, no water, no gas,” said Rita Barros, who has been renting a tenth floor flat at the hotel for the last thirty years. “Now we’re not in a state of war any more but we are under psychological attack. You have a closed building, completely demolished, with only the leftovers. You feel like you’re under threat. You see people being kicked out. That creates an atmosphere.”

There has been very little construction or renovation work done at the Chelsea since March 22, 2013, when a team of building inspectors swarmed the hotel, noting the chipped lead paint, mould, blocked fire exits, leaking pipes and plaster dust. “They stayed here for seven hours, did not leave a dustball unturned, and issued a stop work order for all the violations,” Childs said . “We were dazzled.”

In August, weary of the lawsuits and bad press, Chetrit sold his stake in the Chelsea to his minority partner, King & Grove Hotels. CEO Ed Scheetz promised a renovation more in keeping with the building’s historic character and more respectful of residents. “The prior owner took the position that everyone here was the enemy,” Scheetz told me. “So it was acrimonious and nothing got built. We decided very early on that having a positive relationship with the tenants was the only way to get anything done.”

Scheetz is as smooth as a marble pebble washed up on the steps of this “magical, special, unique” property. As the founder of Hard Rock Hotels, his form inspires scepticism, but he has evidently charmed many of the residents – he claims to have met almost all of them, one on one. A month after taking over, he settled a lawsuit with the tenants association, granting them reduced rent while renovations are carried out and some protection from future increases, in addition to paying their legal fees.

Ed Scheetz in what will soon be the Chelsea’s restaurant.

Under the deal, tenants have a choice between moving to a different apartment in the building – usually a larger space on a lower floor – or being put up at another King & Grove hotel for a month while their flat at the Chelsea is renovated. The hotel desperately needs an overhaul. The radiators in Pennington and Bothwell’s flat were installed in 1910. An electrician told them that the wiring is at least eighty years old.

Building work is scheduled to begin on July 1. The wide, high corridors will get a little narrower, so that pipes for the new heating system and central air conditioning can be hidden. If all goes to plan, the hotel will be open for business by the end of next year – meaning residents will have to put up with eighteen months of drilling and hammering.

Scheetz gave me a second, official tour of the building, starting in the huge old dining room, behind the lobby, with its lion head gargoyles along the beams. He took me up to the roof, which he insisted will remain “a gathering space, for people to enjoy” rather than a bar or club. “I’m sure if you ever could do it you’d make a lot of money, but it’s a fool’s errand,” he said.

We walked over to the Mayan pyramid-shaped turret, home for many years to experimental film-maker Shirley Clarke and soon to host movie junkets, cocktail parties and photo shoots. Across the roof, Julian Schnabel’s old apartment had been abandoned by his daughter Lola when she moved out, leaving nothing but broken mirrors and daubs on the walls.

On the tenth floor, we checked out a huge north-facing room with an arched brick ceiling around four metres high. Metal frames for plasterboard showed where the Chetrit Group intended to divide it in three. “It’s only got one window, so it has to be one room,” Scheetz said. “The building tells you how it has to be laid out. You end up with some of the best hotel rooms in the city.”

When I asked him how much these rooms would cost, he replied: “It’s an iconic property, and I think the product’s gonna blow people away. You never know what the rates will end up being.” In short, it is going to be a very expensive place to stay. At the High Line Hotel, four blocks away, double rooms start at $350 per night.

The Chelsea was built with a mixture of huge apartments and smaller studios on the same floor, in the belief, inspired by French utopian philosopher Charles Fourier, that doing so would promote a healthy ferment by bringing people of varying wealth and social standing together. This novel arrangement, almost unheard of in Gilded Age New York, connected artists and patrons, a tradition that continued through the most storied periods of the hotel’s history. Abbie Rockefeller, meet Arthur Davies. Peggy Guggenheim, meet Jackson Pollock, and so on.

“Philosophically, that’s what we believe in,” Scheetz told me. “We want to have some inexpensive rooms and some very nice rooms, so that people of different ages and demographics can mix like they’re supposed to and historically did.” The question is how much cross-pollination can occur when a tiny room with a view of a ventilation shaft costs upwards of $200, as it inevitably will.

Scheetz promised that the hotel will not have an Arthur C. Clarke suite, with a space motif and a talking computer. There will not be a plaque denoting where Janis Joplin gave Leonard Cohen head in an unmade bed. “We don’t want the Disneyland version of the Chelsea. We don’t want the Hard Rock. We don’t want a museum. We want it to be relevant going forward,” he said. A scheme, curated by a non-profit, that would invite young artists to stay rent free was the only point at which his pitch got a little vague.

The Chelsea’s residents that have made it this far have leases that entitle them to stay until they die. Some have children, and can pass the apartments on. But it seems clear that through a process of attrition, over the years, the number of affordable flats will fall, and the balance with luxury suites will shift. Artists will still stay at the Chelsea, but only after they’ve made it. Successful musicians will pass through, on the label’s dime. Writers will have a coffee on the roof and find somewhere cheaper to live in Brooklyn or Queens.

“The hotel is a living entity like a tree that continues to grow,” Tippins said. “It is a place where you can be anything. It’s a place where you can say anything, write anything, paint anything. This is still a fortress in the middle of this capitalist city where you can experiment in any way that you want to.” But not for long.