The clamour for justice for Chagos Islanders is growing. The conclusion of the 50-year Anglo-American deal to make Diego Garcia (the largest Chagos Island) a US military base will be reached in 2016, and although it is unthinkable that the base will be closed, it provides a chance for islanders to make their case for return – at least to the outer islands.

Once packed into boats by the British government, which called them "migrant workers" and even "a few Tarzans and Men Fridays", the refugees are finally seeing their case taken seriously. From high-profile UK court cases to controversy over US rendition flights and fears nuclear submarines could threaten Africa's atomic-free status, the islands are on the political map.

Britain has pledged to cede the islands, now called the British Indian Ocean Territory, to Mauritius in the vague hope that islanders will be allowed to return. It is easy to see why this option is attractive – it would end Mauritius's bitter complaints about Britain's "illegal" control of the islands and absolve the government of responsibility for the islanders. It would also end Britain's complicity in "serious civil and criminal wrongdoing" by US authorities making use of Diego Garcia.

The problem is the same as the one so callously ignored in the 1960s – the islanders. Despite Britain's appalling treatment, which meant decades of desperate poverty in Mauritius for many, Diego Garcians now living in the UK want the islands to remain British. "We were second-class citizens in Mauritius and if they govern the islands, we will be second-class citizens in our own land," says Allen Vincatassin, the man who champions islanders' rights in the UK.

From a small office in Crawley, West Sussex, a town where around 2,000 islanders and their descendents (about half the worldwide population) now live, he is fighting a desperate battle. "The British government has signalled its intention to cede the territory to Mauritius when Diego Garcia ceases to be a military base, whenever that may be," he says. "Our community does not want this. Mauritius would use these islands for financial gain, for business or as a military base, which would ignore the people.

"We want to be given our right to self-determination. That right is being suspended. Falkland Islanders had their rights protected by this country. The people of Gibraltar were given the right to decide on their future. We want the same right."

Vincatassin, head of the Diego Garcian Society, says Britain must listen to his people: "We are British Indian Ocean Territory citizens, which we are proud to be. We believe we are part of this country. In a normal situation the people would come first but it seems the state of Mauritius comes before the rights of our people."

Vincatassin, who was a small child when he left Diego Garcia, knows his people were wronged by Britain but says some of the damage was repaired when many islanders were given UK citizenship in 2002. "We want the right to return to our homeland, even though many people could not return to that life after so many years in a different world. It's so important to us because the islands are the only ancestral and cultural heritage we have. Without them, I think we are non-existent as a people."

There is no sensible reason why the outer islands, all of which are at least 140 miles from the US base on Diego Garcia, should not be resettled. The only question is who will support this minute island nation – London or Port Louis. It is time we listened to the exiled people of the Chagos Islands. Their answer is clear.