This week, I present to you a whitish building, mostly rectangular, made with large pieces of glass and some nice stone. This may not seem very exciting, especially as works of this description have been the default setting of tasteful British architecture for 20 years. The recent shortlist for designing the relocated Design Museum was made up of purveyors of whitish rectangularity and nice stone, including the winner of the commission, John Pawson. If Inuit are said to have 26 different words for snow, an architecture critic sometimes needs 26 words for off-white.

But this building is designed by architects with a rare sense of those things – relationships, scale, details, nuance, light, matter and pitch – that make a place. It is also in a location, the gardens of Chiswick House in west London, that the chief executive of English Heritage, Simon Thurley, calls "incredibly important" and compares to Stonehenge. Chiswick House helped to change the world or, at any rate, the world of gardens. Created from the 1720s to the 1740s by the wealthy Lord Burlington and his protege, William Kent, it led the way in breaking with the formal geometries of baroque gardens and replacing them with asymmetric and informal patterns that mimicked and followed the shapes of nature. Kent, as Horace Walpole said, "leaped the fence and saw that all nature was a garden". After him came the landscape gardens of Capability Brown and Humphry Repton and, ultimately, every hillock and winding path, and every picturesque gazebo and rockery in suburban gardens everywhere, can claim descent.

The centrepiece of the garden is Chiswick Villa, a domed and porticoed party pavilion created by Burlington and Kent in approximate imitation of Palladio's Villa Rotonda near Vicenza, helping to establish the Palladian style in England. It is hard to think of a more influential work in British architecture and landscape, yet it has been treated negligently for a century or so. The villa has been an asylum for posh lunatics and a fire station, with trucks parked outside. The 17th-century house to which the villa was attached was demolished, to save on maintenance costs.

There was once a risk that the gardens would be submerged under speculative semi-detacheds, but they became a municipal park. This opening up of aristocratic territory is nicely democratic, but it also contributed to the erosion of its original design. Like most British parks, it has suffered since Margaret Thatcher's government decided that spending on open spaces was not a statutory obligation on local authorities. According to Thurley, the gardens became "a big dog lavatory, and a set of targets for youths with spray cans".

Part of the problem was that the villa was the responsibility of English Heritage and the gardens of the London borough of Hounslow, a contradiction of the fact that the "whole point of Chiswick is that the house and gardens were a single entity". A trust has therefore been set up, with the task of managing both together. This has to sustain itself in part through income from events and parties, though £12m from the Heritage Lottery Fund, English Heritage and private donors and sponsors has gone into restoring the gardens and building a new cafe.

The restoration, led by English Heritage, has been a matter of cleaning up, decluttering and unclogging the gardens, as well as restoring its temples and follies. Inopportune fences have been removed and a jumble of different seats and litter bins has been simplified. The sweep from the villa down to an artificial lake has been returned to its original openness and previously blocked vistas have been restored. It is simple, not glamorous, but essential stuff. Later extensions, including a patterned, 19th-century "Italian garden" and a magnificent hothouse, have also been restored.

Then there is the whitish, squarish, nice stone building. This is the cafe, an essential part of the trust's business plan, built at a cost of £1.4m to the designs of the architects Caruso St John, who are leading lights of the generation after Zaha Hadid and the David Chipperfield. It is carefully located, off to one side of the villa, like a fragmentary echo of the wings that Palladio added to his houses. Its position is considered, in the English landscape tradition, in relation to surrounding trees, to the position of the sun and to the outdoor spaces that form around it.

The cafe's style is simplified classical, but with nuances and twists. Its pillars are precisely cut but are of a pitted and pocked Portland stone that resembles the rustic stuff of grottoes. They make high, deep arcades, because, says the architect Peter St John, "the nicest al fresco lunches are in arcades in Spain and Italy". The arcades also allow different kinds of use in different seasons – they can be more or less occupied depending on the weather.

If the original villa was a place for sophisticated townies to party in a contrived version of nature, the cafe is also urbane. Its proportions are more elevated than a typical park cafe, the lamp shades have a surprising mirrored finish. The pillars of the arcade are out of synch with the verticals in the inner wall, which creates unexpected shifts in the interior experience. Sometimes, you feel thoroughly enclosed, sometimes almost at one with the green outside. It nicely captures the best of the spirit of Kent and Burlington: the idea that you adapt, modify and tune the nature that you find, rather than subjugate it.

Terunobu Fujimori's Beetle's House

Meanwhile, at the V&A, you will be able to see a different take on building and nature. This is an installation by Terunobu Fujimori, a Japanese historian who, in his 40s, turned his hand to designing buildings. It is part of the V&A's 1:1 – Architects Build Small Spaces exhibition, in which structures by seven architects are dotted around the London museum.

Fujimori's favourite material is charred wood, a traditional material in Japan, which, if the scorching is done correctly, has properties of endurance and weather-resistance. In a memorable exhibit at the Venice Architecture Biennale, visitors had to enter, stooping, through small, square holes in a blackened screen, the holes being framed in gold leaf. He also likes mud, thatch and wonky tree trunks.

When I meet Fujimori in the south London workshop where his V&A structure was made, the atmosphere is thick with smoke and flecks of ash fall on hair and skin. A rough-hewn stump is being prepared as the base of a little hut to be installed in the museum. Students are doing the scorching and the sticking together – he likes non-professional builders, plus himself, to build his works. Some are stapling bits of charcoal on to ceiling panels to form a decorative pattern. This is not very craftsmanly but Fujimori says that's the point: high degrees of technique would be excluding. Anyone can put up his buildings.

He centres all his structures around a living fire, saying that the origin of building lay in the need to shelter a flame. He also happily admits that his designs, with all their carbonising and burning, are nothing to do with sustainability. They are personal images of the primitive and he does not seem concerned whether anyone else derives satisfaction from them.

My feelings for his work vary between an attraction to his tactile materials and a reaction against the Hobbity quaintness of some of the finished products. Also disappointment in the way he builds – engaging non-experts is all very well, but there's nothing very life-enhancing about chomping at charcoal with staplers.