The Fifa president, Sepp Blatter, has hit back on Twitter against the Manchester United defender Rio Ferdinand as the row over his suggestion that racist abuse could be settled with a handshake intensified and led to renewed calls for him to resign.

Blatter's direct response to Ferdinand came as the sports minister, Hugh Robertson, joined the Professional Footballers' Association chief, Gordon Taylor, in calling on Blatter to quit after suggesting there was no racism in football and that players subjected to abuse should say "this is a game" and shake hands afterwards.

As Blatter desperately attempted to put out the fire surrounding comments made on Wednesday in interviews with CNN and al-Jazeera, on Thursday he responded directly to Ferdinand's trenchant criticism on Twitter.

Ferdinand lambasted Blatter, saying his comments were "so condescending it's almost laughable" and later criticising Fifa for attempting to "clear up the Blatter comments with a pic of him posing with a black man".

In a direct response to the Manchester United defender, the Swiss said: "The 'black man' as you call him has a name: Tokyo Sexwale. He has done tremendous work against racism and apartheid in Africa". He also claimed Fifa had a "long standing and proud record in the area of anti-discrimination".

Sexwale sits on Fifa's Fair Play committee and was one of the former prisoners incarcerated on Robben Island who formed their own football team.

Calls for Blatter to go have intensified since he suggested in an interview with CNN that racism on the pitch was not a problem. "I would deny it. There is no racism," he said. "There is maybe one of the players towards another – he has a word or a gesture which is not the correct one.

Robertson said that Blatter should resign, particularly given his repeated gaffes and wider concerns over corruption and corporate governance. "This is incredibly serious but it is part of a pattern of behaviour," Robertson told BBC Radio 5 Live.

Asked whether Blatter should go, Robertson said: "Yes. I can't see there's anything terribly new in this. We've been saying this for some time. Sepp Blatter will expect the English and the English press to pile in on him."

The Labour leader Ed Miliband also weighed into the row, branding Blatter's comments a "disgrace" and saying that world football needed new leadership.

"I think Sepp Blatter's comments are a disgrace frankly and I think that football needs new leadership. I don't think he has shown in the remarks he has made recently, or indeed his record, that he can provide that leadership for world football."

Taylor was equally critical of Blatter, saying that his latest comments were "the straw that broke the camel's back" and said he hoped the latest storm would lead to his removal from power. He is the leader of world football, he has to be a leader in anti-racism," he told Sky News.

"Racism is divisive and for him to say the player on the receiving end should forget about it at the end of the game and shake hands … we are going backwards … it is not good enough, it's embarrassing.

"And coming on top of his comments which were offensive about female footballers, his homophobic comments about homosexuals not going to Qatar, the World Cup bidding process, he won't have technology over goalline decisions and the corruption which is so plainly evident at Fifa. It is time for him to go.

"He has presided over a lot of issues which haven't been good enough. If he is the leader of world football I am not going to be a follower and I can't believe anyone could follow him.

"It can't be acceptable and I believe it is the straw which broke the camel's back. He should step aside so we can see what Michel Platini can do."

"He has no understanding of what racism is, the ideology behind it, the damage it causes and how it subjugates one group of people as inferior. Enlightened leadership at this level is needed.

"Minor matters on the field often can be resolved with a handshake. Racism is not a minor matter. Kick It Out has no truck with the notion that racism can be dismissed and trivialised in this way."

The 20 Premier League clubs issued a statement following a meeting of club chairmen today, saying: "The English game has been at the forefront of tackling racist behaviour and other forms of discrimination. Everybody in the game in England understands any form of racism is totally unacceptable.

"There are still issues, as there are in society, so with our partners, Kick It Out, the PFA and the FA, we must remain committed and vigilant to maintaining the standards we have set and confronting any incidents that occur."

The former Sheffield Wednesday and Crystal Palace striker Mark Bright criticised Blatter for his "archaic" attitude. He told Radio 5 Live: "This is what Blatter is saying: at Hackney Marshes on Sunday morning you can say whatever you want to your opponent – whatever race, creed or colour he is – and at the end of the game when you shake hands it should be all forgotten – go to the bar and have a drink. Those days are over.

"It would be carnage at every football park across Britain where any ethnic playing in one team against a white person on another team, and [if a player] can say whatever he wants and say 'it's just part of the game'. "It's an old-fashioned view, it's archaic, it's illegal."

Bright added that he did not expect Blatter to stand down from his position, but expressed hope that some of Fifa's corporate backers might be able to exert pressure.

"Will he resign? No he won't. Will he go? Only if the ripple effect takes place, where you've got major sponsors of the World Cup thinking of withdrawing their sponsorship," he said.

"That then would be when Fifa would say they have to act and act quickly."

In his statement following the television interviews, Blatter said: "My comments have been misunderstood. What I wanted to express is that, as football players, during a match, you have 'battles' with your opponents, and sometimes things are done which are wrong.

"But, normally, at the end of the match, you apologise to your opponent if you had a confrontation during the match, you shake hands, and when the game is over, it is over."