"That's as dull as ditchwater," said architect Piers Gough as the organising committee's minibus skirted the copper-clad Olympic handball arena. Amanda Levete, Stirling prize-winning designer, craned her neck to take a look. "Do you think it is deliberately banal?" she asked. "Shocking. The electricity substation is better."

It was a tough start for the architecture of the Olympic park, which was being put to the test by a panel of leading architects and critics convened by the Guardian. Gough, Levete and architectural thinker and historian Charles Jencks were granted access to the £9.3bn menagerie of stadiums and arenas set in former east London wastelands to deliver their first verdict on the UK's most significant public building project since the Festival of Britain in 1951.

How would the buildings match up to the lofty vision of the ideal Olympic city set out by Pierre de Coubertin, the founder of the modern Games? The Frenchman said it should be "steeped in a sort of gravity which need not necessarily be austere and need not exclude joy". He wanted it to attract visitors "on a pilgrimage" and "inspire in them a respect due to places consecrated to noble memories and to potent hopes".

In other words, cathedrals of sport. Handball arena aside, the panel decided London 2012 has provided two of the very best. To the north of the park, the wooden velodrome, where Team GB will hope to repeat their phenomenal gold medal haul in Beijing, seemed tooled to perfection like a Stradivarius violin, Jencks declared. To the south, the swooping curves of the aquatics centre, the three-pool swimming and diving complex designed by Zaha Hadid Architects at a cost of £253m, were "nuts", "fantastic" and "delicious", said Gough.

The walk to the poolside, past artfully placed rows of holstered hairdryers, is an architectural experience in itself. The corridors run at acute angles, a Hadid trademark, and the esoteric geometries of the changing rooms are not what you would expect from a municipal swimming baths. But that is what this facility will become after the Games.

"Oh wow, this is spectacular," said Levete as she entered to see the blue pools glowing under high-wattage lights, the ceiling swooping like a swimmer's dolphin kick and rakes of temporary seating soaring up on both sides. The building is so huge it makes the 50-metre competition pool look like a local swimming bath half the size.

"Without question, the aquatics centre is the star building," she said. "It is a spectacular expression of its sport, resolved in its form and beautifully detailed."

"Bloody hell!" said Gough as he gazed up at the wave-like ceiling. "The way the roof spans all the way is just nuts and fantastic. It is delicious."

It is run close for the title of best building by the £86m velodrome, a swooping structure designed by Hopkins Architects and Expedition Engineering. It has been nicknamed "the Pringle" because of its resemblance to the crisp but, for Gough and Jencks, that seems unfair. "It looks like a bike saddle, or the banking of the track," said Gough.

Jencks admired its pared-down architecture and engineering, saying that with so little that appears redundant, it has a "platonic" shape adding:. "This is the key building for the Olympics," he said.

Spectators are treated to intimate views of the action. The swooping rhythm of the track is echoed by the ceiling. "You can feel the movement of the bikes in the architecture," said Levete. "You can imagine the noise of the tyres on the track."

As bright as they shine, the gems of the velodrome and aquatics centre could not distract from the panel's disappointment at the two most prominent structures in the park, which will fill the TV screens of hundreds of millions of people around the world come the opening ceremony on 27 July.

The 115-metre-tall Orbit viewing platform, the tallest building on the Olympic skyline built with money from Britain's richest man, Lakshmi Mittal, drew pained looks and, according to Gough, gave the impression of having melted in a fire. The London mayor, Boris Johnson, asked Mittal to fund it when he bumped into him in the toilets during the 2009 Davos World Economic Forum.

"It is one of the great disasters borne of mayoral desperation," said Gough of the design by Anish Kapoor and Cecil Balmond. "It looks like all the steel they saved on the stadium has been fragrantly wasted on a sculpture. It is so painful, because Anish Kapoor is a very special sculptor."

But it was the lack of architectural ambition in the £431m main stadium that caused most concern. Has London missed a huge opportunity, they asked.

"It's an Ikea stadium," said Gough. "It is sparse, direct and unfussy … We expect stadiums to take your breath away and unfortunately this one doesn't. But when it is full of people it will go 'whumph!' and the atmosphere will completely overwhelm the architecture anyway."

"I find it unremarkable and a little bit disappointing," said Levete as she scrutinised its white zig-zag of struts from across the park's wide gravel concourse. "Beijing's Bird's Nest stadium was the icon for the Olympics in China and that is what we are missing. It is not going to capture anyone's imagination. Why did we go for an off-the-shelf design when we could have had anyone in the world design it?

"Images of this building are going to be broadcast around the world. It doesn't hold the iconography of the moment. This was the building they needed to spend money on. They should have found the money to do it." The lack of spending was a calculated decision by the publicly funded Olympic Delivery Authority. The stadium design that helped to win the Games in 2005 was a sinuous structure that looked like a ring of rippling muscle dreamed up by an avant garde firm called Foreign Office Architects (FOA). It looked beautiful, but also dangerous.

At the time, Wembley stadium was costing more and taking longer than anybody ever expected, so the FOA designs were quietly dropped and contractor-led consortiums were asked to bid. That meant design would almost certainly take second place to the imperatives of cost and delivery. The brief was also limiting, because it called for an 80,000-seater that could be converted to 25,000 seats afterwards. That pleased the International Olympic Committee, which was desperate to shed its reputation for foisting white elephant arenas on host countries such as Greece and Australia. "We were going to be the sustainable Games, the cost-effective Games and it was a totally different brief," said Ian Crockford, the stadium's project director.

Such was the threat of political interference in such a high-profile project that many builders and architects considered it too risky to even bid, and in the end the authorities were left with only one consortium that met the criteria – the team that had just built Arsenal's Emirates stadium with great efficiency – architects Populous, engineers Buro Happold and builder McAlpine.

The spare structure they produced for the London Olympic stadium wins admiration for its simplicity and economy. With its exposed steelwork, simple flat bowl shape and lack of decoration, it avoided any political embarrassment that would have been caused by rows about the value for money of design flourishes or cost overruns. But it is, Levete said, "painfully pragmatic".

Home, for the 17,000 competitors and officials, will be the athletes' village, which transforms a little piece of the London borough of Newham into the kind of apartment block suburbia you find on the outskirts of continental European cities. The balconies on one block even have patterns stencilled on them that look like towels hanging out to dry as if in Portugal or Spain.

"It is the culmination of Richard Rogers's vision towards Britain as a European city: 10-storey, good buildings, very egalitarian," said Jencks.

The village, with an academy school for 1,800 students and a health centre, will be one of the legacies of the Olympics. Some of the £1bn budget has gone on well-finished public spaces. The pavements are made from beautiful natural stone and semi-mature trees have been craned in. In a nod to the optimism of postwar Britain, the school has been designed in an updated 1950s style, using a rotunda and glass the colour of patinated copper.

"This is like a design from the previous London Olympics in 1948 or the Festival of Britain," said Jencks.

Indeed, the Olympics planners had the 1951 festival in mind when they started looking for different approaches to the whole project, said Kevin Owens, head of design at the London Organising Committee. "It was the last time we as a nation embarked on something of this national importance."

The Olympic park is unified by a tangle of waterways, lawns and thousands of newly installed plants and trees. The parkland will expand once the Games are finished and Gough said he expects it will be the most attractive part of the scheme.

The architectural styles on show against this backdrop, while all resolutely modern, do not suggest a clear theme. "I am a little surprised at the lack of coherence as an ensemble and it is quite difficult to understand the links between them," said Levete. "But we have two good buildings and a disappointing stadium. That's enough. The velodrome and the aquatics are memorable and are world-class buildings."