Recent studies are finding that drugs such as LSD and psilocybin can help to alleviate depression, anxiety and addiction—and may have profound things to teach us about how the mind works, writes Michael Pollan.

To anyone who lived through the 1960s, the proposition that psychedelic drugs might have a positive contribution to make to our mental health must sound absurd. Along with hallucinogens like mescaline and psilocybin (that is, magic mushrooms), LSD was often blamed for bad trips that sent people to the psych ward. These drugs could make you crazy.

So how is it possible that, 50 years later, researchers working at institutions such as New York University, Johns Hopkins, UCLA and Imperial College in London are discovering that, when administered in a supportive therapeutic setting, psychedelics can actually make you sane? Or that they may have profound things to teach us about how the mind works, and why it sometimes fails to work?

Recent trials of psilocybin, a close pharmacological cousin to LSD, have demonstrated that a single guided psychedelic session can alleviate depression when drugs like Prozac have failed; can help alcoholics and smokers to break the grip of a lifelong habit; and can help cancer patients deal with their “existential distress” at the prospect of dying. At the same time, studies imaging the brains of people on psychedelics have opened a new window onto the study of consciousness, as well as the nature of the self and spiritual experience. The hoary ‘60s platitude that psychedelics would help unlock the secrets of consciousness may turn out not to be so preposterous after all.

The value of psychedelic therapy was first recognized nearly 70 years ago, only to be forgotten when what had been a promising era of research ran headlong into a nationwide moral panic about LSD, beginning around 1965. With a powerful assist from Timothy Leary, the flamboyant Harvard psychology professor, psychedelics had escaped the laboratory, falling into the eager arms of the counterculture. Yet in the decade before that there had been 1,000 published studies of LSD, involving 40,000 experimental subjects, and no fewer than six international conferences devoted to what many in the psychiatric community regarded as a wonder drug.

Compared with other psychoactive compounds, these powerful and mysterious molecules were regarded as safe—it’s virtually impossible to overdose on a psychedelic—and nonaddictive. Rats in a cage presented with a lever to administer drugs like cocaine and heroin will press it repeatedly, unto death. LSD? That lever they press only once.

This is not to say that “bad trips” don’t happen; they do, especially when the drugs are used carelessly. People at risk for schizophrenia sometimes have psychotic breaks on psychedelics, and people surely do stupid things under the influence that can get them killed. But the more extreme claims about LSD—that it scrambled users’ chromosomes or induced them to stare at the sun until blind—were debunked long ago.

It wasn’t until the 1990s that a small band of researchers began to unearth what an NYU psychiatrist describes as “a buried body of knowledge” about the therapeutic potential of psychedelics. Perhaps the most promising application of the new drugs was in the treatment of alcoholism. Few people in Alcoholics Anonymous realize that Bill Wilson, the founder, first got sober after a mystical experience he had on a psychedelic administered to him in 1934, or that, in the 1950s, he sought, unsuccessfully, to introduce LSD therapy to AA.

In parts of Canada during the 1950s, psychedelic therapy became a standard treatment for alcoholism, and a 2012 meta-analysis of the six best-controlled trials of LSD therapy for alcohol addiction during that period found a “significant beneficial effect on alcohol misuse.” Early studies of psychedelics for the treatment of several other indications, notably including depression and anxiety in cancer patients, also showed promise.

These first-wave studies were, by contemporary standards, poorly controlled. That’s why many of the early experiments are now being reprised using more rigorous modern methods. The early results are preliminary but encouraging: A pilot study of psilocybin for alcohol dependence conducted at the University of New Mexico found a strong enough effect to warrant a much larger phase 2 trial now under way at NYU.

Another recent pilot study, at Johns Hopkins, looked at the potential of psilocybin to help people quit smoking, one of the hardest addictions to break. The study was tiny and not randomized—all 15 volunteers received two or three doses of psilocybin and knew it. Following what has become the standard protocol in psychedelic therapy, volunteers stretch out on a couch in a room decorated to look like a cozy den, with spiritual knickknacks lining the bookshelves. They wear eyeshades and headphones (playlists typically include classical and modern instrumental works) to encourage an inward journey. Two therapists, a man and a woman, are present for the duration. Typically these “guides” say very little, allowing the journey to take its course, but if the experience turns frightening, they will offer a comforting hand or bit of advice (“trust and let go,” is a common refrain).

The results of the pilot study were eye-popping: Six months after their psychedelic session, 80% of the volunteers were confirmed to have quit smoking. At the one-year mark, that figure had fallen to 67%, which is still a better rate of success than the best treatment now available. A much larger study at Hopkins is currently under way.

When I asked volunteers how a psilocybin trip had given them the wherewithal to quit smoking, several described an experience that pulled back the camera on the scene of their lives farther than ever before, giving them a new, more encompassing perspective on their behavior.

Recent trials involving psilocybin mushrooms have been found to help alleviate depression and break the grip of addiction.

“The universe was so great, and there were so many things you could do and see in it that killing yourself seemed like a dumb idea,” a woman in her 60s told me. During her journey she grew feathers and flew back in time to witness various scenes in European history; she also died three times, watched her soul rise from her body on a funeral pyre on the Ganges, and found herself “standing on the edge of the universe, witnessing the dawn of creation.”

“It put smoking in a whole new context,” she said. It “seemed very unimportant; it seemed kind of stupid, to be honest.”

Matthew Johnson, the psychologist who directed the study at Hopkins, says that these sorts of “duh moments” are common among his volunteers. Smokers know perfectly well that their habit is unhealthy, disgusting, expensive and unnecessary, but under the influence of psilocybin, that knowledge becomes an unshakable conviction—“something they feel in the gut and the heart.” As Dr. Johnson puts it, “These sessions deprive people of the luxury of mindlessness”—our default state and one in which addictions flourish.

Perhaps the most significant new evidence for the therapeutic value of psychedelics arrived in a pair of phase 2 trials (conducted at Johns Hopkins and NYU and published in the Journal of Psychopharmacology in 2016) in which a single high dose of psilocybin was administered to cancer patients struggling with depression, anxiety and the fear of death or recurrence. In these rigorous placebo-controlled trials, a total of 80 volunteers embarked on a psychic journey that, in many cases, brought them face to face with their cancer, their fear and their death.

“I saw my fear…located under my rib cage,” a woman with ovarian cancer told me. “It wasn’t my tumor, it was this black mass. ‘Get the f— out,’” she screamed aloud. “And you know what? It was gone!” Years later, her fear hasn’t returned. “The cancer is something completely out of my control, but the fear, I realized, is not.”

Eighty percent of the Hopkins cancer patients who received psilocybin showed clinically significant reductions in standard measures of anxiety and depression, an effect that endured for at least six months after their session. Results at NYU were similar.

Curiously, the degree to which symptoms decreased in both trials correlated with the intensity of the “mystical experience” that volunteers reported, a common occurrence during a high-dose psychedelic session. Typically described as the dissolution of one’s ego followed by a merging of the self with nature or the universe, a mystical experience can permanently shift a person’s perspective and priorities. The pivotal role of the mystical experience points to something novel about psychedelic therapy: It depends for its success not strictly on the action of a chemical but on the powerful psychological experience that the chemical can occasion.

Few if any psychiatric interventions for anxiety and depression have ever demonstrated such dramatic and sustained results. The trials were small and will have to be repeated on a larger scale before the government will consider approving the treatment. But when the researchers brought their data to the FDA last year, the regulators reportedly were sufficiently impressed to ask them to conduct a large phase 3 trial of psilocybin for depression—not just in cancer patients but in the general population.

So how does psychedelic therapy work? And why should the same treatment work for disorders as seemingly different as depression, addiction and anxiety?

In one pilot study, smokers got two or three doses of psilocybin. Six months later, 80% were confirmed to have quit smoking.

When scientists at Imperial College began imaging the brains of people on psilocybin, they were surprised to find that the chemical, which they assumed would boost brain activity, actually reduced it, but in a specific area: the default mode network. This is a brain network involved in a range of “metacognitive” processes, including self-reflection, mental time travel, theory of mind (the ability to imagine mental states in others) and the generation of narratives about ourselves that help to create the sense of having a stable self over time.

The default mode network is most active when our minds are least engaged in a task—hence “default mode.” It is where our minds go when they wander or ruminate. The Imperial scientists found that when volunteers reported an experience of ego dissolution, the fMRI scans of their brains showed a precipitous drop in activity in the default mode network, suggesting that this network may be the seat of the ego.

One way to think about the ego is as a mental construct that performs certain functions on our behalf. Chief among these are maintaining the boundary between the conscious and unconscious realms of the mind as well as the boundary between self and other.

So what happens when these boundaries fade or disappear under the influence of psychedelics? Our ego defenses relax, allowing unconscious material and emotions to enter our awareness and also for us to feel less separate and more connected—to other people, to nature or to the universe. And in fact a renewed sense of connection is precisely what volunteers in the various trials for addiction, depression and cancer anxiety trials have all reported.

This points to what may be the most exciting reason to pursue the new science of psychedelics: the possibility that it may yield a grand unified theory of mental illnesses, or at least of those common disorders that psychedelics show promise in alleviating: depression, addiction, anxiety and obsession. All these disorders involve uncontrollable and endlessly repeating loops of rumination that gradually shade out reality and fray our connections to other people and the natural world. The ego becomes hyperactive, even tyrannical, enforcing rigid habits of thought and behavior—habits that the psychedelic experience, by loosening the ego’s grip, could help us to break.

That power to disrupt mental habits and “lubricate cognition” is what Robin Carhart-Harris, the neuroscientist at Imperial College who scanned the brains of volunteers on psychedelics, sees as the key therapeutic value of the drugs. The brain is a hierarchical system, with the default mode network at the top, serving as what he variously calls “the orchestra conductor” or “corporate executive” or “capital city.” But as important as it is to keep order in such complex system, a brain can suffer from an excess of order too. Depression, anxiety, obsession and the cravings of addiction could be how it feels to have a brain that has become excessively rigid or fixed in its pathways and linkages—a brain with more order than is good for it.

Dr. Carhart-Harris suggests that, by taking the default mode network offline for a period of time, psychedelics can, in effect, “reboot” the brain, jog it out of its accustomed grooves and open a space for new pathways to arise. His lab has made maps of the brain’s traffic patterns on psychedelics showing that, when the default mode network is quieted, myriad new connections spring up in the brain, linking far-flung areas that don’t ordinarily talk to one another directly.

The value of such an experience is surely not limited to the mentally ill. There are rich implications here for what one psychedelic researcher calls “the betterment of well people.” Who doesn’t sometimes feel stuck in destructive habits of thought? Or couldn’t benefit from the mental reboot that a powerful experience of awe can deliver?

One of the lessons of the new research is that not just mental illness but garden-variety unhappiness may owe something to living under the harsh rule of an ego that, whatever its value, walls us off from our emotions, from other people and from nature. “For the moment,” wrote Aldous Huxley, describing his own psychedelic journey in 1954, “that interfering neurotic who, in waking hours, tries to run the show, was blessedly out of the way.”

This essay is adapted from Mr. Pollan’s new book, “How to Change Your Mind: What the New Science of Psychedelics Teaches Us About Consciousness, Dying, Addiction, Depression and Transcendence,” which will be published on May 15 by Penguin Press. His previous books include “Food Rules,” “In Defense of Food” and “The Omnivore’s Dilemma.”