Seychelles, a group of 115 islands off the east coast of Africa with 92,000 residents, does not figure prominently on many lists, but it leads the world in locking people up. It is the only country with a higher incarceration rate than the United States.

If the reforms recommended in a new report from the Brennan Center for Justice were fully implemented, the U.S. would fall from second to fourth place on that list—behind Seychelles, St. Kitts and Nevis, and Turkmenistan, but still far ahead of every other liberal democracy, not to mention Iran, Saudi Arabia, and Zimbabwe. The report is nevertheless an admirable effort to grapple with the morally and fiscally pressing question of who belongs behind bars and who doesn’t.

Between 1974 and 2007, the U.S. imprisonment rate (excluding people in local jails) soared from 102 to 506 per 100,000, thanks to changes in sentencing (including mandatory minimums and “three strikes” laws), parole (including “truth in sentencing” laws), and prosecutorial practices (including an increased tendency to bring charges). Since 2007 the imprisonment rate has declined a bit, but it is still more than four times as high as in the mid-1970s.

This imprisonment binge was largely a response to crime rates, which rose dramatically from the 1960s until the early ’90s, when they began a long slide. Today the violent and property crime rates are half what they were in 1991.

Although “it is tempting to look at these data and assume that mass incarceration caused this decline in crime,” the Brennan Center says, research suggests imprisonment played a modest role that shrank over time. It turns out that increases in the number of people behind bars and the amount of time they spend there yield diminishing returns.

In recent years, states such as California, Texas, New York, and New Jersey have seen crime rates continue to fall while substantially reducing their prison populations. The trick is figuring out which offenders can remain free and which prisoners can be released without compromising public safety. Prison should be reserved for the most serious offenders, and longer sentences are not necessarily better, especially in light of evidence that they do not enhance deterrence and may actually increase recidivism.