'At first I said no," says Renzo Piano. "We were very busy. For me, the idea of building a convent next to Le Corbusier at Ronchamp was, in any case, a bit crazy." Certainly, it must have felt like a big risk. The chapel of Notre Dame du Haut at Ronchamp is one of the 20th century's most treasured buildings, and Le Corbusier a demigod in the architectural firmament; being asked to build alongside this French national monument, an international destination for religious and cultural pilgrims, is like receiving an invitation to knock up a postmodern extension to the Parthenon or St Peter's in Rome.

But then Piano met Sister Brigitte de Singly at his studio in Paris, caved in and said yes. The architect was busy with towering commercial projects such as Shard London Bridge, at 310metres [1,017ft] Europe's tallest building, as well as the expansion of Boston's opulent Isabella Stewart Gardner Museum, both due to open next year. Meanwhile, Sister Brigitte and her nuns were hoping to leave their home of 800 years in Besançon, in order to be closer to Le Corbusier's chapel.

With an all-in budget of £9m, at least 60 times less than that of the Shard, the convent for the Clarisses, or Poor Clare Sisters, was to take up a disproportionate amount of his time over the next five years. Funding was a slow and complex process; the money was realised through local government funding, charitable and religious donations, and the sale of the nuns' former convent.

A nun speaks with a woman in the dining room. Photograph: Sebastien Bozon/AFP

When I meet the architect and the abbess, lunching frugally with pilgrims and builders at trestle tables set on a wooded hill, below Le Corbusier's chapel, I can see why Piano said yes. "If Sister Brigitte was to be my client, then what else could I say?" Piano says. "She has a profound love of architecture, of landscape, of sacred space – and even of people without religion, like me. She wanted a place of silence and prayer. I said: 'I can't help you with prayer, but perhaps I can help with silence and a little joy.'"

Just as Le Corbusier's chapel was created for a Catholic church he did not believe in, and shaped by a very particular interpretation of the medieval monasteries he never lived in, so Piano has produced a building of quiet refinement and spirituality at Ronchamp. "Sister Brigitte reminded me of the need for quiet, for nature, for slowness, for simplicity," he says. "She reminded me of the long tradition architects have had of working with the church."

The nunnery is, for the most part, invisible – or will be when new trees have been planted, and plants have spread over the concrete roofs Piano and his Building Workshop have half-buried in the hillside. "Landscaping is half the project," Piano says. Even so, the project met great opposition when plans were unveiled three years ago. The Fondation Le Corbusier, a fierce guard of the architect's reputation, was quick on the attack. "They began to scream: you can't do this!" says Piano. At the time, the foundation's director Michel Richard argued: "We are trying to make sure the site is preserved for eternity. We are afraid that in 10 years, the sisters will go away and they will be replaced by a B&B."

"Of course, they were worried that we wanted to build too close to Le Corbusier," says Jean-François Mathey, who, with Sister Brigitte, has been the driving force behind the project. Mathey is president of the Association de l'Oeuvre Notre Dame du Haut, the organisation that commissioned the chapel from Le Corbusier 60 years ago. "In fact, they didn't want anything new built here." When Piano announced his plan to hide the building away in the hillside, Jean Louis Cohen, the distinguished French architectural historian and board member of the Fondation, told the press: "Maybe you wouldn't see it, but you would feel it."

Renzo Piano (right) speaks with a priest in the chapel. Photograph: Sebastien Bozon/AFP

All of this is understandable, but Mathey had been thinking about a new religious foundation for Ronchamp for some time. "The chapel is a great attraction to believers, to cultural tourists, to architects, to anyone with a soul," he explains, "and we have 100,000 visitors a year. But we didn't want [it] to become only a tourist attraction, or a funfair; we wanted to make sure it stays a place of prayer." When Archbishop Luigi Ventura, the papal envoy to France, comes to bless the convent on 2 October, Ronchamp will be reconsecrated in the hearts of the Catholic faithful.

And despite the Fondation's fears, Piano has made a great improvement to the hilltop site. A grim concrete visitors' centre that had lurked between car park and chapel has been demolished. A new visitors' centre, dug into the hill, forms the base of the convent. There is a bookshop and a gallery behind a welcoming zinc-and-glass facade; in winter, a roaring log fire set behind a glass screen will greet those who have battled with snow and fog to get here.

Above is the convent proper. This wraps itself around contours of the hill, burrowing into the landscape like the strands of a rosary pressed gently into the earth. The strings of the rosary are the convent's corridors; its beads are the rooms leading off them. The crucifix at its centre is the chapel, the Oratory.

On one side of a simple central entrance, a long corridor lined with sweet-smelling, floor-to-ceiling cedar cupboards leads to the nuns' cells and living quarters. There is room for just 12 Poor Clares. Aside from their life of prayer and work, they will look after visitors seeking more than architecture and landscape can offer.

A nun of the order of St Clare reads in her room designed by Renzo Piano. Photograph: Sebastien Bozon/AFP

The cells are spare, calm and chastely beautiful. They are no more than 2.7 metres square, but have custom-designed timber furniture, warm orange walls, superb natural lighting and stirring views south and west to the valley below. The rooms are fronted by private winter gardens, glazed suntraps serving as architectural gaps, or pauses, between inner and outer worlds. (They will also help keep the cells warm in winter, cool in summer.)

Every light switch, every chair

The refectory is gathered around three sides of a courtyard, with glazed walls but open to the sky. It must be wonderful to eat here as the rain or snow falls. At the heart of the convent, the chapel's concrete vault curves in two different planes, like the upturned hull of a boat (an image of the Church as a ship of souls), while a concealed slit in the chancel wall facing the hillside brings a halo of daylight into its deepest recesses. "Architecture," as Le Corbusier said, "is the masterly, correct and magnificent play of masses brought together in light." Piano's work here is quietly masterful, built around a minimal palette of concrete, timber and zinc; the fact that he and his team have designed and crafted every last detail, from chairs to light switches, within such a modest budget is a minor modern miracle. Buried into the hillside, the convent should prove cheap to heat and light. Deep bore holes bring warmth from the ground, while daylight is reflected through the building at every turn. It felt comfortable here on the intensely humid day I came to visit.

"I have tried to make it like a little hill town," Piano says. It's an appropriate analogy. Between 1922 and 1935, Le Corbusier planned new city centres (which were never realised), inspired as much by medieval monasteries as by modern life. "I have found the solution to workers' housing," he wrote to his parents in 1907. "I saw, in the harmonious countryside of Tuscany, a modern city crowning the top of a hill. The ring of monks' cells formed the noblest silhouette on the landscape. Each cell overlooks the plain and opens at a lower level into a small, enclosed garden. I thought I had never seen such happy living arrangements."

The pilgrimage chapel at Ronchamp and the monastery of Saint Marie de la Tourette at Eveux-sur-Arbresle, north-west of Lyon, were Le Corbusier's last great buildings, both built on shoestrings. Piano has many more buildings in him, and yet it is fascinating to see this thoughtful architect nurturing one of his most considered buildings on a low budget, for nuns living and praying at the foot of Le Corbusier's chapel. He was absolutely right to have said yes.