Much of American Samoa, with its population of about 55,000, is a South Pacific paradise. It’s also a kind of netherworld, where those born in the U.S. territory don’t have the rights enjoyed by the vast majority of Americans.

A three-judge panel on the District of Columbia court of appeals ruled Friday that those born on American Samoa are not entitled to sue to become U.S. citizens. A complaint filed on behalf of several Samoans, some of whom live there and others who live in the United States proper, pointed out that those born in American Samoa are considered U.S. nationals, but not citizens. That means that even when they move to Hawaii or another state, they’re not entitled to vote, hold office or enjoy certain other rights reserved for citizens. Their U.S. passports have a restriction that labels them as U.S. nationals.

“Today marks the first time a federal court of appeals has ruled that citizenship by birth on U.S. soil is not a fundamental right,” Neil Weare, who argued the case for the Samoans, said.

The panel, made up of three of the circuit’s most conservative judges, relied on law made in the Insular Cases decided in 1901. Those decisions said those living in lands taken by the United States as booty from the Spanish-American War were not eligible to become citizens. Supreme Court Justice Henry Brown referred to the inhabitants of those territories as “savage” and “alien races” in his opinions, according to Pema Levy of Mother Jones. That’s not surprising, since Brown is also the author of the infamous Plessy vs. Ferguson decision that enshrined segregation into law. What is surprising is that such a decision based on racist principles could be used to make law in 2015.

The suit did not have the backing of the American Samoan government. Officials there oppose automatic citizenship, fearing it might harm local traditions and possibly force the restriction that reserves land ownership to Samoans to be removed.

As it is now, Samoans must live in a U.S. state for at least three months, and then apply for naturalization.

Those who filed the case may now petition for it to be heard by the full D.C. Circuit. Those judges, who are more liberal, might reverse the decision of the smaller panel.