If we are to believe that fascism and racism are not connected then you must also believe I am writing this as an utterly devoted football fan. It is an absolute travesty that I am not paid a seven-figure sum to come on television and do the post-match analysis: "Well, at the end of the day they needed to get more goals." But I must accept that even I don't have the extensive knowledge of football cliches, honed over many years, that the pundits do.

Of course, many folk I know love football. They love watching it, playing it and talking about it. Some have won scholarships because of their skills. Some are women; they are passionate beyond belief. For it is a belief system and one that the football bigwigs exploit. The Premier League is where the problems lie; not with five-a-sides out in the cold in the park. For how long do the managers and owners and shareholders and hacks cover up the wrongdoings that we can all see?

Really, I have to ask what is wrong with a smidgeon of fascism when we have had decades of sexism (both domestic violence and rapes covered up) or racism or out-and-out homophobia? Having read many of the defences of Paolo Di Canio being given the Sunderland job, I see what we are now expected to accept: Fascism-Lite.

It is quite remarkable that one can claim to be fascist and not racist. Do I need to spell it out? Di Canio ("compulsive", "excitable")has not only said he is a fascist but has given fascist salutes – an action that is actually banned in many European countries. To the untrained eye they look like Nazi salutes. Oh, but silly me, it's just part of his allegiance to Lazio and is a Roman salute – which just happens to be where many believe the Adar/Hitler salutes derives from anyway.

Still, Di Canio has said in the past that he was fascinated by Mussolini (compulsive, excitable, Italian!), who of course never said anything racist, who never helped deport Jews and who never used racism as part of his ultra-nationalism! Today he has declared that he did not support "the ideology of fascism".

Right now in Greece with the Golden Dawn, and as we have seen on the football terraces of eastern Europe, fascism is alive not as an interesting little hobby but as brutal as ever. It must be real enough for the fundamentally useless Sepp Blatter to have banned its open support on the pitch.

To think that an ultra-nationalist authoritarian ideology is NOT racist requires turning several blind eyes. But then the hypocrisy here is that no one cared when he managed Swindon except the GMB union, who admirably withdrew sponsorship. I don't know how many big sports writers covered this at the time. David Miliband's flounce is gesture poltics. He is off anyway.

Yet again though we have many arguing: "Let's focus on football, not politics." No! Look at yourselves. This was said when Sheryl Gascoigne was in a sling with a black eye and Gazza was allowed to play. And a lot of good it did him. When women's refuges tell us domestic abuse rises on match day, no one cares. When young women are herded into parties as part of the celebration for winning teams – while the Wags stay at home – it is considered perfectly OK.

Porn may have started it but football certainly made mainstream the concept of "spit-roasting", the homoerotic act where a woman has sex with more than one man at the same time. She becomes the conduit between men, yet of course the idea of openly gay footballers is still one many struggle with.

Clearly many ordinary decent fans are appalled by all this. They have done much to tackle racism and antisemitism. Now we can see rascism moving into the changing room and certain fans recruited into the EDL.

This is not about destroying the evident pleasure the sport provides but there is something rotten at its core. These huge amounts of money and sponsorship deals have led to an amoral culture. Obviously moneyed testosterone-central needs to be kept in check to continue to provide joy to millions.

The Olympics showed us a different model: superfit, lowlier but disciplined sports people whom I very much doubt were not at it day and night after competing. And who can blame them? What happens with football, though, is that it is elevated into a unverse where normal rules don't apply. In this universe that is only about "the game", we see the madness of declaring Fascism-Lite is justifiable.

This is intolerable.

Football may occupy a special place in our culture and in many hearts. But while it continues to kick women, black players and gay people in the face while justifying Nazi salutes, how can it be a beautiful game? It is an ugly, ugly business.