Zac Goldsmith's embarrassment about his non-domiciled tax status is more important than it seems. It cannot be dismissed as simply being an effective blow landed by the Lib Dems in their campaign to retain a key seat; underpinning that blow are a much wider range of issues that relate to fundamental injustice and even illegality within the UK tax system.

The concept of domicile, like so much in UK tax law, has no legal definition. Your domicile is, in effect, your natural home. It is not your place of citizenship, or your ethnicity, or even where you live: it is the place to which you owe your long-term affiliation. To put it another way, your domicile is the place you consider to be your place of national origin.

The concept as used in UK law is, at its core, racist. It was of considerable value in the colonial era. When there were no passports and a quarter of the world was pink on the map, domicile made clear who you were; part of "Blighty", or not, as the case may be.

And, in a very real sense, that remains the case – except the tables have turned. Because domicile is a concept quite separate from tax residence (itself a concept in need of radical reform), the trick now is to be tax resident in the UK, but non-domiciled. That way, you get all the advantages of living here, but don't have to pay all your taxes for doing so. Only your UK-source income and gains, and those income and gains you bring to the UK from abroad are subject to UK tax if you're non-domiciled.

This is, of course, only of benefit if you have non-UK-source income and gains. For the vast majority of those temporarily resident in the UK – for example, the 3.8 million current non-UK-born UK employees representing 12.9% of the UK workforce – the domicile rule will be irrelevant. Their only earnings will arise in the UK, and if they are involved in remittances, they will be sent from the UK, not to it. But for a small minority of about 100,000 people, the rule provides something quite different: a unique advantage to structure their affairs so that they can pay very little or no tax in the UK, bar an annual membership fee for joining the non-dom club of £30,000 per annum, introduced in 2008.

Whether Goldsmith is exploiting this situation is not the real question. The real questions are fourfold. First, why do we let an elite who are as resident in the UK as anyone else pay less tax than others who are also resident here? Second, why do we allow non-domiciled status to be claimed by people who are born here, have lived here much of their lives and are so integrated into UK society that they are even MPs and peers here? Surely, better policing is needed when the loss to the UK from this rule is, in my current estimate, about £3bn a year? Third, why do we allow the UK to continue to operate as a tax haven in this way, at considerable cost to our international credibility and at cost to the credibility of the anti-tax haven campaign the UK is spearheading? And finally, and most importantly, why do we tolerate a tax law that is illegal?

As I have argued for some time, the terms of the Race Relations Act 1976 and the Race Relations Act (Amendment) Regulations 2003 make clear that unlawful indirect race discrimination takes place in the UK if a public authority provides a service that affords a person of one national origin a social advantage over a person of another national origin, unless there is a legitimate and proportionate objective that justifies that different treatment. The granting of non-domicile status is the provision of a service by a UK public authority and it does confer considerable advantage on those who are granted it without there appearing to be any legitimate and proportionate reason for doing so.

Those who lose are, of course, UK-domiciled people who cannot enjoy the tax advantage that non-doms have and which society could most certainly not afford to grant to all of us. The fact that it is the majority who are being discriminated against does not stop this being an abuse of the law, as it has been since national origin become a grounds for discrimination in 2003.

As Richard Wilkinson and Kate Pickett have shown, the more unequal a society is, the less successful it is. The UK domicile is about creating inequality on the basis of national origin – itself an illegal act. What better reason to get rid of it as an economically and ethically unjustifiable anachronism from a bygone age, exploited now only by the richest in our society so that they can get richer at cost to all the rest of us?

And knowing that this is the case is, no doubt, the cause of Goldsmith's rightful embarrassment. We should save him his blushes: let's abolish the rule, now.