Is the right to vote a fundamental human right? You could be forgiven for assuming that the answer is yes, given the way that so many modern politicians talk about it. Yet the phrase "right to vote" does not occur in the universal declaration of human rights adopted by the United Nations in 1948. Nor is it to be found in the European convention on human rights of two years later.

These are interesting and surely significant omissions. Somehow, it is hard to believe that the exclusion of such a resonant individual entitlement as the right to vote from two such landmark documents can be entirely accidental.

But its absence from them is a reminder that voting rights are an issue not just of human rights but also of political power. And this may help to explain why, in this era of supposedly universal human rights, the idea that prisoners might be allowed to vote triggers such outrage and resistance among so many British politicians of all parties.

To be fair to the UN declaration, while it does not speak of a right to vote in the same fundamentalist sense that it talks about a right to life, it does say that elections "shall be by universal and equal suffrage". This goes much further than the European convention, which only gets round to the subject of elections in its additional protocols, in which signatory states pledge to ensure "the free expression of the people in the choice of the legislature".

Ironically, however, it was under this much vaguer and more guarded protocol that the European court of human rights found in 2005 by 12 votes to five that John Hirst, a British prisoner serving a sentence for manslaughter, possessed a right to vote. As the minority judgment made clear, this was a bit of a stretch. Nevertheless, the court's ruling required the British government to act. It called time on existing UK electoral law, which has a blanket disqualification on all serving prisoners from voting in elections.

Both the last government and the current one have tried to wriggle off the hook and to delay their legal obligation to act on the Hirst judgment.

Politically, this one is a no-brainer. Not only are there are no voters in prisons, there are no votes in prison reform either, and the tabloid press gives no quarter on the issue. Public opinion overwhelmingly opposes giving prisoners the vote, with 76% against and only 17% for in a recent YouGov poll.

Legally, the issue is pretty clear too. But it points in the opposite direction. Governments, like citizens, should obey the law once a case has been decided. As a founding signatory to the convention, Britain is obliged to act on the basis of the court's rulings. There is, as the cabinet office minister Mark Harper told MPs last month, "a legal obligation".

Philosophically, however, the case for allowing prisoners to vote is very powerful. Prisoners are human beings. They are sent to prison as punishment not for punishment. They remain citizens in all respects save loss of liberty. They possess legal rights and human rights in other respects. So why deny them the vote? Sixteen European countries allow all prisoners to vote in elections; Britain and Ireland are the only ones in Europe with a long and rooted tradition of democracy that deny prisoners the vote. The US, with its legal and political roots in English law, shares this culture of disenfranchisement.

The reluctance to let prisoners vote rests on a view of rights that sees the vote as a privilege. It goes back to John Locke and beyond. Locke argued that human rights existed in nature, but that the need to form well-governed states involved a contract that allowed properly constituted governments to place restrictions on natural rights. The criminal, almost by definition, has no part in this contract.

The objections of the politicians of 2011 to the Hirst ruling are full of echoes of Locke long ago. "Civic rights go with civic responsibility," the current attorney general, Dominic Grieve, said last year, "but these rights have been flagrantly violated by those who have committed imprisonable offences." Giving the vote to serious offenders was not "moral or even decent", added the former shadow home secretary David Davis this week.

But it depends on your morality. If your morality is that of Locke's era, the vote is a privilege that comes with having a stake – a financial stake – in society. In that era, there were not merely no votes for prisoners but no votes for women, for the homeless, for the property-less, for the non-tax-paying or for the illiterate either. Yet few would argue today that a homeless, illiterate or jobless person – let alone a woman – should not be entitled to vote. Our view of voting is based on an approach of rights not privileges. Criminals are, of course, different from the homeless or the jobless. But they are all adult human beings. In today's morality – and increasingly in today's law too – human rights are inherent in all human beings, irrespective of everything from gender to wealth. So why, on grounds of political principle rather than political pragmatism, deny rights to prisoners?

For all its many problems, modern imprisonment no longer implies the medieval notion of "civil death". Prisoners no longer forfeit their property or their rights when the gates close behind them. However imperfectly it may operate, our penal system sees prisoners as citizens behind bars, who can rejoin society and lead good and full lives. As human beings, prisoners are like you and me. They have inalienable human rights. And one of those rights, in the modern world, should be the right to vote.