Swift action is required to "crush" any return of racism to football before it impacts on wider society, David Cameron said on Wednesday. The prime minister said he had seen for himself the influence that behaviour on the pitch could exercise over young children and others off it.

But, speaking at the opening of a Downing Street anti-discrimination summit with former players and football leaders, he expressed confidence that the good work of two decades in battling prejudice in the sport would not be lost.

"If everyone plays their role, then we can easily crush and deal with this problem," he said. "We will not let recent events drag us back to the bad old days of the past."

He told the panel, including the former England players John Barnes and Graeme Le Saux, that he often took his young son to local football matches and had seen the impact the game had. "What happens on the field influences what happens off the field. You see children as young as six imitating the behaviour they see on the field. So this is not just important for football, it's important for the whole country."

Cameron said there were many "extraordinary role models" among professional footballers but that problems with racism had "crept back in".

"We want to make sure football is all about a power to do good, rather than anything else," he said.

The event to promote a more "inclusive" sport comes amid fears that high-profile incidents are harming football's image. This month Liverpool's Luis Suárez was forced to apologise for refusing to shake hands with Manchester United's Patrice Evra before a game. Suárez had only recently returned to action following a ban for racially abusing Evra during a game in October.

The England defender John Terry is due to face trial over the summer accused of racially abusing QPR's Anton Ferdinand. Terry denies the charges.

The culture secretary Jeremy Hunt said he had given the FA two months to come up with a firm action plan to tackle a variety of race and homophobia-related issues.

He said the talks had been "productive" and had focused on issues such as the lack of black coaches, managers and referees and the use of social media such as Twitter to broadcast racist views relating to football.

Hunt also said there was a need to address the culture in dressing rooms themselves - "how we draw the line between banter and offensive language".

Amal Fashanu, who recently made a documentary about homosexuality in the sport, 14 years after her footballer uncle Justin hanged himself, was among those attending.

Cameron said the UK should be proud of the "fantastic achievement" of the campaign to rid the sport of racism over the past two decades – noting that other countries had failed to take similar action. "I hope what we can agree today is to make sure that everybody who has the ability to deal with this issue takes the steps they can. I am sure we can crack this problem."

That included the government, clubs and other football bodies, he suggested.

The FA chairman David Bernstein said the FA was working from the grass roots up to deal with the issue and was determined not to allow any complacency because of the past success. The governing body had already shown its willingness to deploy "very tough sanctions where necessary", he said – calling for action across the board from fans to players to managers, chairmen and authorities.