From the tracks of the High Line – the derelict elevated railway on New York's Lower West Side currently being transformed into a mile-and-a-half-long "park in the sky" – this most bustling of cities seems suddenly quiet and still. Three storeys above the hubbub of Chelsea – where a few days earlier people had queued round the block to vote for Barack Obama – even the perpetual horns of taxi cabs seem muted, and the only human voices heard are the occasional shouts from workmen. The Hudson river flows gently out towards the skyscrapers of Jersey City. Golfers thwack balls towards the huge nets of Chelsea Piers.

Up on the tracks, the concrete path that will form the backbone of the new park splits off in places to weave between seed beds, recalling the plant life that sprang up between the tracks during nearly 30 years of disuse. Elsewhere, between new wooden benches and young beech and pine trees, stretches of the steel track have been restored to their original positions. At two points during this first section of the line – due to open in spring – the route ducks right through huge gaps in private buildings where goods trains used to deliver directly to the upper floors of warehouses. The park will end at a new public plaza in the fashionable Meatpacking District, overlooked by a new outpost of the Whitney Museum of American Art.

The High Line has attracted embarrassingly lavish praise since the project to save and renovate it began almost a decade ago. In 2001, Adam Gopnik of the New Yorker wrote of the then-overgrown and under-threat tracks: "The High Line does not offer a God's-eye view of the city, exactly, but something rarer, the view of a lesser angel: of a cupid in a Renaissance painting, of the putti looking down on the Nativity manger." But Friends of the High Line, the campaign group that saved the line from demolition and is now in charge of rebuilding it, seems to be seeking a simpler reaction from the public, something closer to photographer Joel Sternfield's verdict upon seeing the tracks for the first time: "It's green! It's a railroad! It's rural! Where am I?"

The High Line was constructed in the early 1930s as part of New York's West Side railway. Train tracks had originally been built at ground level along the middle of 10th Avenue, but this proved so dangerous that the stretch became known as "Death Avenue", and trains had to be preceded by a man on horseback, nicknamed the West Side Cowboy, waving a red flag as a warning to pedestrians.

The High Line was the solution. A mile and a half long and 30ft above street level at its highest point, it stretched from the Meatpacking District on the Lower West Side up to West 34th Street, level with the Empire State Building, covering 22 blocks in total. At certain points the tracks disappeared into the upper floors of warehouse buildings in order to deliver meat, milk or manufactured goods, reappearing again on the other side.

But as the freight industry switched from rail to road, use of the High Line declined. Part of it was pulled down in the 1960s, and the last train ran in 1980, carrying "three boxcars full of frozen turkeys" for Thanksgiving, according to actor Ethan Hawke, who narrates a video about the line on the Friends of the High Line website.

In the years that followed, residents and businesses in Chelsea lobbied for the disused tracks – now overgrown with weeds and plants – to be torn down. In 1999, two locals, Joshua David and Robert Hammond, met at a community meeting and formed the idea of trying to save the High Line and turn it into a "park in the sky". "We looked for other organisations working to save the structure but nobody was," says David.

A former magazine writer, David was working on an article about the changing face of Chelsea when he first noticed the line. "It just captured my imagination. It seemed like an amazing opportunity to create something that would allow people to experience the city in an utterly new way. To go and tear it down without considering how we could use it seemed a waste to me."

After forming Friends of the High Line, David and Hammond persuaded the rail company that then owned the site to take them up on to the tracks. "It was extremely exciting," says David. "We went up through one of the old factory buildings. I remember stepping out on to the tracks, and there's an incredible 13-block-long stretch immediately: this incredible vista of wild grasses and flowers. It took my breath away."

With the help of celebrity backers such as Hawke, fellow actor Edward Norton, and designer Diane von Fürstenberg, FHL managed to overturn mayor Rudy Giuliani's demolition orders and get the new administration of Michael Bloomberg behind the project.

The group held a design competition for the refurbishment or reuse of the tracks, and proposals – including a rollercoaster and a mile-and-a-half-long swimming pool – were publicly displayed at Grand Central station. The winning design, by Field Operations and Diller Scofidio + Renfro, was chosen in 2004, and resembles a long walkway above the streets, flanked by lawns, flowers and trees, with numerous benches and access points via stairs and escalators.

Groundbreaking for the new High Line took place in April 2006, with Bloomberg in attendance as well as New York senators Hillary Clinton and Chuck Schumer, and the first stretch of the line is set to be completed early next year, with the second finished in 2010.

The line runs through former industrial neighbourhoods now defined by flats, restaurants, nightclubs, art, and fashion, and this area is being transformed by the project. The 2005 re-zoning that approved the park also allowed for new construction alongside the track and last year New York magazine identified an already-emerging "High Line neighbourhood", complete with "glittering retail spaces and restaurants and condos", as well as High Line-themed restaurants and festivals.

David says the credit crunch has had some impact on all this building work. "There was a flurry of building starts at the time the High Line was approved; there are certainly fewer things starting now. But there is still an extraordinary amount of activity: cranes all over the place and numerous buildings under construction.

"We hoped it would attract architects and developers who would create extraordinary and original buildings along it – and that's really come to pass." David hopes the High Line will not only be used by people strolling to work or eating lunch, but will also become "a real destination for visitors to New York … a cultural destination". Craig Schwitter of Buro Happold, the lead engineer on the project, suggests it could host open-air markets, concerts and other artistic events.

The project that began with an attempt to save the elevated wilderness that Alan Weisman, in his book The World Without Us, identified as an example of nature taking back the urban landscape has now morphed into something rather different: a regeneration project for an entire neighbourhood. But David is happy with that. "It's at once a preservation project about the city's past and a construction project about the city's future," he says. "The High Line has always evoked the passage of time, with the incredible landscape that has sprung up on top of it."

He had wanted the new design to retain some of that original, unique landscape, "but there was no way it could be walked on without being destroyed. The challenge for the design team was to do that and allow the public to come up in a safe and accessible way." To that end, the plan uses plant material "pushing up through walkways" and reintroduces railway tracks in a number of locations. At two locations the new High Line will go through "doughnut holes" in private buildings, just like the old freight trains did when the buildings were warehouses. "It will be quite clear to visitors how it worked," says David.

David, 45, and Hammond, 39, a former painter and consultant, now run Friends of the High Line full-time. The group, which has a staff of 16, is itself now transforming – into a "conservancy", an organisation that will manage and maintain the park in partnership with New York City council. David and Hammond are currently trying to raise both the $50m to build the park ($22m has been raised already) and an unspecified sum to maintain it and pay for staff once it has been completed.

Yet above 30th Street, where the line snakes around the West Side rail yards, FHL is still the campaigning group it once was. This section of the line is still under threat and could be torn down as part of the redevelopment of that area. "That's something we are fighting very strongly," says David. That part of the line has not yet been designed, although the basic plan is to connect it to the Hudson River Park at 34th Street.

New York is a city of spectacular views from towering vantage points. But according to David the High Line is different. "It's called the High Line, but it's not that high. It's three storeys off the ground. I think that's what's so special. It's not that experience on top of the Empire State Building where everybody is like ants or little toys. You can shout down to people on the street. It's connected to street life but at a distance. There's a sense of remove."

"There used to be a lot of elevated rail in the city," notes Schwitter, "but that was all removed after world war two." The High Line restores that perspective, he says. "A quieter perspective, somewhere to walk quietly, and observe the city."

New York architects Diller Scofidio + Renfro, best known for the city's beloved High Line park, are putting the public at the centre of their work with imaginative projects that defy conventional notions of 'building', writes Rowan Moore