The call by the New York Times last week for the legalisation of marijuana is an important milestone in a two-year trend that has seen seismic shifts in both the reality and public perception of America's 40-year war on drugs.

At both the federal and state level, several reforms have taken place to reduce both the intensity of narcotics law enforcement and the severity of drug sentencing. These developments warrant attention as much for the progress they reflect as well as for the unfinished work they reveal.

Since the war on drugs has historically targeted black Americans most, these changes will disproportionately affect the black community and this is cause for celebration. But beyond select reforms, crucial questions remain about how the nation got to this point and how the causes of such racism in public policy may be addressed.

For Americans, race has always been the elephant in the room. On lands streaked by the blood of genocide, America built a nation on the backs of African slaves. But it didn't stop there. From the opium laws of the 1800s, which imposed racial control over Chinese migrant workers, to the eugenics movement of the 20th century, which made dangerous theories of racial superiority state and national social policy, to the internment of Japanese Americans during the second world war, the nation has long struggled to reconcile the lofty, humanist rhetoric of her founding with the practical realities of American life.

Throughout America's history, official and unofficial systems of racial oppression have arisen, been challenged, and then gone underground, shape-shifting themselves to return another day. In the modern era, as lawyer Michelle Alexander has argued in her book The New Jim Crow, the drug war stepped in to become the latest system. In 1971, as the gains of the civil rights movement for black Americans and other minorities might have seemed to usher America into a post-racial age, the drug war renewed the nation's commitment, however subtly, to the obstruction of black progress.

Today as the marijuana economies in Colorado and Washington begin to take flight, Alexander noted the inescapable undertow of race that continues to haunt this moment of apparent progress at play: "Forty years of impoverished black kids getting prison time for selling weed, and their families and futures destroyed … Now, white men are planning to get rich doing precisely the same thing."

Over those four decades, the war on drugs has failed abjectly in its stated mission – addressing a legitimate concern about rates of US addiction – but succeeded overwhelmingly in what would appear its de facto goals – making drug crime the primary preoccupation of law enforcement, flooding the courts with drug cases and overcrowding prisons with the world's largest population of inmates, more than 50% on drug-related charges. Taken together, these accomplishments have produced a system of mass incarceration that costs taxpayers an estimated $51bn a year, becoming one of the nation's leading employers. Within its walls, black Americans represent more than 50% of those sentenced for drug crimes, despite the fact that black people represent only 13% of the population and do not use drugs more or less than white people.

A decade ago, when I began investigating the drug war in what would become my documentary The House I Live In, acquaintances were intrigued. They knew I was neither a drug user nor a dealer. They also knew that I was a comfortable white American, and thus highly unlikely to have been affected by the drug war personally. Inevitably, the question would arise about whether I was an advocate of marijuana legalisation, which had then become a primary focus for most reformers.

I responded always with indignation, saying that I did not support legalising marijuana if that meant simply giving dreadlocked white snowboarders easier access to weed. Rather, I was concerned with the drug war's implications for poor and minority Americans, whose communities had been ravaged by the war's destructive machinery. I also saw a philosophic error in separating marijuana from other drugs.

Part of what is assumed by advocates of the drug war is that the government has a legitimate role determining what substance an adult can choose to put in his or her body in the exercise of the right to life, liberty and the pursuit of happiness. Arguing that one drug should be legalised while others not seems to elide this question of public policy. And this elision is dangerous, first because it enables the country to avoid a deeper dialogue about the illegality of drugs per se and, second, because I feared it could let steam out of the debate about the drug war more broadly, reducing public pressure for its overhaul. Worse, I even feared that by going easy on weed we would tighten the screws on the rest, keeping the system and its predations intact.

I've since changed my mind on the importance of marijuana as a target for reformers, owing to what I've learned about the role it plays in driving the cycle of personal, family and community destruction on which the war thrives. "Gateway drug" has been the term often used by drug warriors to suggest that, with one puff of a joint, a young person may find himself hurtling down a road to hard drugs. Despite this notion's popularity, it has little or no basis in science. Yet marijuana is a gateway drug for countless young Americans into a lifetime of involvement with the criminal justice system. For many, an arrest for possession at a young age can start a chain reaction that leads first to drastically reduced employability and then to a higher likelihood of becoming engaged in the underground economy of drug distribution, often the only job available. Once this happens, it becomes almost a fait accompli that that person will spend a serious portion of his life rotating in and out of the system.

The numbers speak volumes. Of the 2.2 million prisoners serving in the US, nearly 25% were convicted of marijuana possession. Legalisation, were it retroactive, would dramatically reduce prisoner numbers while profoundly stemming the tide.

Such numbers have converted me into a devout advocate for removing marijuana's illegal status. Yet while I have come to favour this, I eschew the term "legalisation" and prefer a "tax and regulate" approach not just to marijuana but to all illegal drugs.

In my view the time has come for the US government to take a responsible health approach to the drug-addiction problem while honestly owning up to the dangerous social forces that produce and foster such addiction. While I laud the Times and others for coming to the view that illegality for marijuana and other drugs is perhaps not the answer, a tax and regulate approach would (as it does with alcohol) assign to the government its proper role – not of intervening in the relationship between an adult and their body but rather in protecting certain members of society from others and protecting society as a whole from the dangers of unregulated substances. Along the way, the government would naturally charge an appropriate tariff for such activities, the proceeds of which could go a long way not only to offering crucial drug counselling and treatment services but to relieving the racial, societal and economic pressures that produce the preconditions for drug abuse in the first place.

Eugene Jarecki is a New York-based writer and film-maker. His Grierson, Emmy and Sundance-winning works include Why We Fight, The Trials of Henry Kissinger and The House I Live In