Soldiers have been drafted in to fill empty seats at the London 2012 Olympics after prime seating at the aquatics centre, gymnastics arena and basketball venue again went unused on the second day of competition.

More than 50 soldiers took seats at the North Greenwich Arena on Sunday morning when they were left empty by the "Olympic family". Some of the troops, working at the Olympics to provide security, said they were scheduled to start shifts but instead were offered courtside seats at the basketball arena to watch the USA v France.

Despite featuring superstars including Kobe Bryant and being close to a sellout with the public, there were around 40 empty seats in the arena reserved for Olympic and sporting officials. "We're seat fillers," said one of 15 soldiers drafted on Sunday afternoon. "They asked who likes basketball and we put our hands up."

The London organising committee (Locog) refused to rule out offering seats to G4S staff if the problem persists. The action came after the culture secretary, Jeremy Hunt, said on Saturday the empty seats were "very disappointing" and Locog launched an investigation. Organisers indicated international sporting federations are the main cause of the problem in the accredited areas, and the International Olympic Committee has told them to deal with the issue.

Pictures of athletes competing against banks of empty seats were beamed around the world on the first day of competition but Lord Coe, the Locog chairman, was initially dismissive of the problem, saying on Sunday: "Those venues are stuffed to the gunnels."

He denied that having troops fill seats appeared shambolic. "If we have the army sitting there on rest periods we can ask them if they want to sit in there and watch it," he said. "We take it seriously. I don't want to see swathes of those seats empty."

Coe said he believed the problem would settle down, adding that it was a result of representatives of national Olympic committees, sporting federations and some sponsors taking time to "work out the shape of their day" and decide where to go.

The organisers faced growing pressure to act. The former Olympics minister, Tessa Jowell, called for more tickets to be handed out to soldiers, school children and teachers "to ensure that every single available ticket is offered to somebody who will use it". Lord Moynihan, the British Olympic Association chairman, suggested a "30-minute rule" for non-attendees, with empty seats to be filled by home fans. He said organisers "owe it to the British sporting public to give them an opportunity to attend one of the most historic sporting events of their lives".

Coe dismissed the idea. On Sunday morning there were an estimated 2,000 empty seats at the basketball match between Nigeria and Tunisia with 70% of a section allocated to sponsors, Olympic officials and athletic federations, vacant. More than 300 seats at the morning's gymnastics were also unfilled and around 2,500 were empty during the heats at the Aquatics Centre. Other venues were close to full – including for table tennis, judo and boxing.

The Ministry of Defence said the troops would be off duty and take up of the seats would be voluntary. Up to 150 local school children and school teachers who are already accredited to the Olympic Park will also be deployed under a pre-existing scheme meant to fill vacant seats in accredited areas.

"It's completely wrong to say this is a sponsors issue," said Mark Adams, spokesman for the IOC. Sponsors Visa, Coca Cola and McDonalds said the vast majority of their tickets were being used. As well as inviting troops, Locog said it had taken 900 gymnastics tickets from allocations to accredited individuals and sold them to the public. Excluding the football, between 100,000 and 120,000 Olympic tickets remain available, it said. A Wimbledon-style returns system, allowing people already on the Olympic Park to pick up tickets to the last parts of sessions of basketball, hockey, handball and water polo has also been launched. Stewards will ask people leaving venues after the first match of a two-match session if they are willing to let someone use their ticket for the second game. The ticket will be scanned and reissued at box offices by the Orbit, the hockey arena and velodrome. Childrens' tickets cost £1 and adults £5. Around 280 were sold on Saturday.

During the ticketing process, in which 6.6m of the 8.8m tickets were made available to the public, organisers repeatedly highlighted the lengths they were going to make sure there were no empty seats. They said they wanted to avoid the situation that occurred in 2008 at Beijing and at other Games, where even at sessions that were officially sold out there were large areas of empty seats – particularly for morning heats.

In the vast majority of the 2012 venues, at least one-fifth of seats are reserved for sponsors, officials, the media and the "Olympic family". In the most popular sessions, such as the opening ceremony and the 100m athletics final, the proportion is closer to half. Those thronging the park who had been unable to get into the venues to watch any live action were critical of the empty seats.

"It's infuriating to see so many empty seats on TV," said Ed Shorthose, a London-based father of two who has been trying for months to get tickets to see the Games. "Surely it can't be beyond the organisers to allow real sports fans to fill them up on a first come first served basis?"

The Locog chief executive, Paul Deighton, has talked in the past of his determination to avoid any empty seats, limiting the length of sessions and holding seminars with sponsors to underline the importance of only taking tickets they can use. He introduced a so-called "fans in front" policy to try and ensure vibrant atmospheres in every venue.