When Lords Foster and Rogers unveiled their designs in 2006 for two dramatic skyscrapers that would rise up out of the ashes of Ground Zero, it looked like New York had finally found a way to bounce back up after the disaster of 9/11.

Norman Foster hailed the moment as marking "the renaissance of the New York skyline". Richard Rogers said his practice had designed a "transparent and legible building".

Less than three years on, Britain's two leading architects appear to be caught at the centre of increasingly acrimonious wrangling between the public owner of the site and the private developer in charge of construction. The buildings, which would both rise to more than 70 storeys, are in danger of being squashed to mere "stumps" of four or five floors.

The Port Authority, which owns Ground Zero, has made it clear that it wants a radical rethink. This is partly in recognition of the dire economic conditions and partly to ensure that the core of the reconstruction – the largest tower, formerly known as Freedom Tower and a memorial to the almost 3,000 people who died in the 9/11 disaster – is finished on time. Its latest plan, leaked to the New York Daily News, would make the Rogers and Foster towers the fall guys.

The Foster tower, originally conceived as 79 storeys and 390 metres (1,270 feet) high, topped by four gleaming diamonds, would be reduced to what the paper calls "a glorified, prettied-up stump". The Rogers building, 71 storeys and about 350 metres, would also become "another stumpy building" of no more than five floors.

With no end in sight to any of the main building projects more than seven years after the 2001 attacks, the Ground Zero saga is becoming a greater embarrassment with every day that passes. A project that had been intended to show America's resilience in the face of external threats has come to illustrate its bureaucratic and commercial weakness.

Squabbling has reached such a pitch that New York's mayor, Michael Bloom­berg, has stepped in and called a summit at his Gracie Mansion for later this week. "Now more than ever, we – the public and private sectors both – have to come together to maintain our resolve and get it done," Bloomberg said.

In the middle of the dispute stands the Port Authority, a joint body of New York and New Jersey states. In addition to owning the site, it is directly building the largest and most important of the new skyscrapers, Tower One, which will rise to a symbolic 1,776 feet. It was called Freedom Tower, though that name has quietly been dropped for fear of dissuading potential tenants from taking up office space.

Tower One is already under construction. Its frame has reached almost 20 storeys high and can just be seen rising above the blue plastic that surrounds the 16-acre site – a source of some relief for many ­people despairing of any progress at the site.

The other main party to the dispute is Larry Silverstein, the private property magnate who leased the Twin Towers shortly before they were attacked. He has been given the right to develop the three smaller towers conceived for Ground Zero.

They include Tower Two – the Foster design; Tower Three – Rogers's building; and Tower Four, a 64-storey structure by the Japanese architect Fumihiko Maki. Tower Four, the least ambitious of all the elements intended for the site, is already being built and is to some extent shielded from the current crisis.

Silverstein can in theory press ahead with the building of the Rogers and Foster towers, but in practice he is out of cash. He is down to the last $1bn (£660m) of the $4.5bn insurance money he received for the Twin Towers.

In a statement, the Silverstein group insisted it was fully committed to replacing the "soaring office towers destroyed eight years ago. The Silverstein team has not wavered on rebuilding the World Trade Centre, and we never will."

But that assessment is widely considered to be grossly optimistic. Silverstein, hit by the credit crunch, recently asked the Port Authority for help in financing the towers, and was rebuffed.

The authority is already having to find more than $3bn to build the former Freedom Tower, and with both New York and New Jersey states already dealing with huge budget shortfalls there is little appetite for bailing out a private developer.

The final blow came last month when the authority commissioned an outside consultancy, Cushman & Wakefield, to advise it on how much demand there would be for the new office space created by Silverstein's three towers. The consultants concluded that the Foster building would not fill its floors until 2026. The Rogers skyscraper would not be fully let until 2037.

The Rogers partnership, Rogers Stirk Harbour, is still working on its design for Tower Three despite the visible slowing in activity. Foster and Partners completed its design in 2008 and is ready to go on site at any time.

But the truth is that prolonged delay seems inevitable for the British-designed towers, leaving two huge gaps in the original masterplan for Ground Zero like missing teeth.