New figures have revealed that the number of people playing sport regularly has fallen in the past year, despite unprecedented public investment and the legacy promises attached to the 2012 Olympics. The sports minister, Hugh Robertson, said the newly released statistics were "very disappointing".

While some sports, including boxing, table tennis and cricket, showed an increase in participation, others including swimming and tennis have seen numbers continue to fall.

Sport England, the grassroots sport funding agency that invests around £250m of lottery and taxpayer money every year, also admitted the figures from its Active People survey were disappointing. It said economic factors were increasingly cited as a reason for playing less sport and claimed the numbers had been hit by bad weather last winter.

While the number of people playing sport three times or more a week has risen modestly since 2007-08, Sport England said it was of real concern that the number of young people and women playing sport regularly had gone down over that period.

The government recently formally dropped an earlier pledge made by the Labour government to increase the number of people playing sport three times or more a week by one million between 2007-08 and 2012-13. The shadow Olympics minister, Tessa Jowell, recently insisted that she did not regret setting the target while in government and has bemoaned the decision to axe it. The former sports minister Richard Caborn has said the vision for a sports participation legacy from the 2012 Games is in danger of failing completely.

The latest figures show the number of people playing sport three times or more a week has increased by only 111,800 since 2007-08. The total now stands at 6.927m, which is a decline of 11,200 on the same time last year. The only four sports to show a statistically significant increase in once-a-week participation since 2007-08 are boxing, table tennis, mountaineering and athletics.

Over that period, a total of 19 sports have seen their numbers shrink. Swimming has declined from 3.24m to 2.81m, while tennis has gone down from 487,500 to 375,800.

The figures also showed that the number of people doing no exercise at all in the previous month had risen to 57.8% of the population.

Robertson said: "Although not unexpected, these figures are very disappointing. It is for this reason that we have spent the second half of this year working with Sport England and governing bodies on a new strategy with particular emphasis on youth sport, that we will announce in the new year."

The new strategy, covering the next investment period of the so-called Whole Sport Plan through which £480m was invested in governing bodies between 2009 and 2013, is expected to target specifically the drop-off in young people playing sport when they leave school. The figures show that the number of 16- to 19-year-olds playing sport three times or more a week has declined from 930,400 to 825,900 since 2007-08.

As well as encouraging more people to do more sport, another of the pledges made on behalf of the London Games was to inspire young people through sport. The Sport England chief executive, Jennie Price, said that public investment in grassroots sport could suffer if governing bodies did not do more to increase participation.

"I thought that by now we would be seeing a stronger upward trend. I am really disappointed we're not growing. We have to be really clear about finding out why and how we address it," she said. "In the reasons people are giving, economic factors are coming through more strongly than in the past and are now one of the biggest factors. If the sports want to grow participation, they've got to have a really good low-cost offer."

The Guardian revealed earlier this year that the government planned to move to a more performance-based approach for funding grassroots sport, in line with the "no compromise" ethos adopted by elite funding body UK Sport. That could see some major sports receive major funding cuts, though Price said she expected the overall level of investment to remain around the same.

"We are getting much tougher with them and they do now understand that if they perform really badly over a long period of time, they will lose money. There are some bigger sports that are getting the message. What we're not getting is enough growth fast enough. Football, for example, has got a really good, credible programme but it's just not driving its numbers. And that's not good enough."

Clive Efford, the shadow sports minister, said: "At a time when participation should going up as we approach the Olympics today's figures on participation are very disappointing.

"This government has been a disaster for sport from day one – cutting over 60% of funding from school sports is not the way to increase future participation in sport and deliver a lasting legacy from the Olympic Games."