A lot of us loved the World Cup a bit less by Thursday night. The bond we thought would survive all shocks and violations slackened. Without the store of glowing memory we might feel like letting go.

Old world arrogance is not the love wrecker. As Fifa delivered their double coup of Russia (2018) and Qatar (2022) football's communal carnival was cast as the private possession of 22 plutocrats. The World Cup has been stolen: appropriated by unaccountable empire builders who pick it up and drop it across the world for reasons that have nothing to do with custodianship and plenty to do with Fifa gain.

Have I been cryogenically frozen for the last 30 years, you cry? Is this news? Well, yes. The "football family" has never been one you would be glad to see moving in next door. The world governing body long ago mutated from administrating to deal-making as federations and their continental clusters snatched at the vast new wealth from television deals and commercial "partnerships."

But this is something else. This is Fifa demanding detailed technical reports and then ignoring them. This is Russian political influence and Qatari petro-wealth smashing aside all considerations of fairness and fan participation in favour of hidden agendas. None is harder to fathom, by the way, than Geoff Thompson, England's representative in Zurich, whose glassy passivity was so aptly juxtaposed by the conniving all around him.

The frontier principle was established 16 years ago, of course, with the US World Cup. That was my first as a reporter. I can remember the accusations of selling out, of subservience to corporations, in a land where the 0-0 draw was an affront to Darwinian American values.

Compared to the Russia and Qatar votes, that successful foray into the world's most powerful country looks in retrospect like a festival of idealism and innocence. No one should blame Russia or Qatar for wanting the World Cup (who wouldn't?). The point is that Fifa have turned it into a version of Bernie Ecclestone's F1 circus. But the World Cup is not theirs to sell. It's ours. It belongs to everyone and no one.

There would be no whining here had 2018 gone to Spain-Portugal or Holland-Belgium. English pique is not at play. My big hope is not that the World Cup will pitch up in Plymouth one day but that English youngsters will be taught to play like Xavi, Mesut Ozil or Andrés Iniesta.

France, in 1998, was a return to old Europe, with glorious results; Japan and South Korea – another expedition into a vast emerging market. Germany, 2006: old Europe again, post-Berlin wall. Next: Africa's first World Cup, and a fresh surge of unease at Fifa's control-freakery and hypocrisy.

The Zurich crowd were embracing Africa. The township child was joining the nice Swiss family. All this, while Fifa and Uefa fines for racism in grounds remained derisory. In South Africa itself the hosts were obliged to build vast new arenas to fit Fifa's lavish specifications. In Durban, a new cathedral rose within throwing distance of the perfectly adequate Absa Stadium.

A year on – in June – you can expect to read plenty of dispatches from Rustenburg and Cape Town about high maintenance costs and under-used venues. The Fifa spaceship comes down then it rises six weeks later stuffed with profits. The invasion of sovereign territory by Swiss lawyers armed with Fifa dictats is another scandal. Increasingly, Sepp Blatter and his pals pick countries who are desperate to stage the World Cup for prestige and infrastructure transformation. These needy states are easier to push around.

At least it will be fun watching Fifa tell Russia what to do. Suppose – just imagine – the Russians did a deal with some Fifa members. For the next eight years Putin's Russia will be the boss in that relationship. In Zurich conspiracy theories were more plentiful than pralines. "The loss of England's World Cup bid to Russia is the result of Russia not playing fair," announces Dr Yuri Felshtinsky, who wrote "Blowing up Russia" with Alexander Litvinenko, the murdered former Russia spy. Felshtinsky alleges that with its KGB background Putin's government has a history of exerting pressure on sports mandarins.

A question nagging at me is how Panorama came into possession of documentary proof of Fifa corruption in the 1990s just weeks before the 2018 ballot. Quite a coincidence. Russia, among others, were bound to benefit from Fifa paranoia about journalistic scrutiny.

To have to think like this has become necessary, and onerous. Next time Arsène Wenger says the Champions League is a better competition and that international football is dying it will be harder to feel indignation's sap rising in the throat. With a failing team and a dead bid, English spectators have plenty of good reasons to disengage.

What is the World Cup meant to mean? The shirts, the fascination with each country: the buttercup yellow of Brazil, the dark brilliance of Argentina, the new Spain, English ineptitude, French mutinies, the excitement of pinning up a wall-chart, camper van tours, making new friends, watching games in bars in the host nation and feeling a small part of the unfolding narrative. This is the World Cup – not Fifa. One day we will take it back.