Some people swear that regular sessions of acupuncture help relieve their back pain and headaches. And now there’s evidence they may be right.

In a study published in the Archives of Internal Medicine, researchers led by Andrew Vickers, an epidemiologist and biostatistician at Memorial Sloan Kettering Cancer Center, report that acupuncture is effective in reducing people’s chronic pain — more so than standard pain treatment and slightly better than using sham needles, suggesting that the benefits of real acupuncture are due to something more than the placebo effect.

The findings counter those of the last large study on the subject, which found that the needle technique was no better than a fake acupuncture treatment — using random pricking with toothpicks — in reducing people’s pain. But Vickers says his meta-analysis of the data, in which researchers reviewed 29 previous studies involving 17,922 participants, does a few things the previous studies did not. For one, he and his colleagues began by looking at only the most rigorous trials involving acupuncture and pain relief — those that directly compared acupuncture treatment with some type of sham needle therapy in which needles were either inserted only superficially or placed in locations that are not known by acupuncture standards to be key treatment points in the body. The authors of the analysis contacted each of the researchers on the previous studies to discuss with them how they separated the two treatment groups. By limiting their review to the most robust studies published, the authors could assess with more confidence acupuncture’s true effect on participants’ reports of pain before and after treatment.

Next, rather than simply summing up the total effect of acupuncture reported by researchers in previous studies, Vickers’ team asked for their raw data on individual participants’ self-reports of pain. Not all scientists use the same scale for reporting results, which makes it difficult to compare and consider such measurements of pain as a whole. With the raw data, Vickers and his team were able to standardize the participants’ responses and compare them in a more meaningful way.

The result was a clear and “robust” effect of acupuncture in relieving chronic pain in the back, neck and shoulders, as well as pain due to osteoarthritis and headaches, Vickers’ team found. Compared with people undergoing sham needle treatments, those receiving acupuncture reported drops in back and neck pain of 0.23 standard deviations, and of 0.55 standard deviations compared with those not using acupuncture at all. On a pain scale of 0 to 100, that meant that among the participants, who started out with an average baseline pain score of 60, pain ratings fell to 30 on average for those who got acupuncture, 35 for those who received fake acupuncture, and 43 for people who got usual care and no acupuncture.

“The effects of acupuncture are statistically significant and different from those of sham or placebo treatments,” says Vickers. “So we conclude that the effects aren’t due merely to the placebo effect.”

Acupuncture is an ancient Chinese remedy for a curing a host of chronic ills, from headaches to back pain and menstrual cramps. The practice, like all medical traditions from the East, is built on the concept of maintaining the balance of various body elements — including blood and nutrients along with less measurable ones like the energy force known as chi. Inserting needles at designated point on the body is supposed to intercept or unblock the flow of such elements, and lies at the heart of the centuries-old therapy of acupuncture.

However, these theories are completely foreign, even weird, to Western medicine, which has a harder time accepting unquantifiable entities such as chi. That’s why Western researchers have struggled not only to document objective evidence of acupuncture’s effectiveness, but also to provide some hints about how it may work.

Some doctors say the needles may release endorphins, the pleasure-inducing, painkilling chemicals that saturate the brain and numb pain signals. But such theories can’t fully explain why acupuncture patients say their chronic pain episodes become less frequent and less intense over time, with regular, long-term sessions. Some say the benefits of acupuncture are purely in the mind, a psychological placebo effect. But either way, for many pain patients, acupuncture does provide palpable relief.

Asks Dr. Andrew Avins in a commentary accompanying the study, if the treatment works, does it really matter whether the effect is physiological or psychological? “At least in the case of acupuncture, Vickers et al have provided some robust evidence that acupuncture seems to provide modest benefits over usual care for patients with diverse sources of chronic pain,” he writes. “Perhaps a more productive strategy at this point would be to provide whatever benefits we can for our patients, while we continue to explore more carefully all mechanisms of healing.”

In other words, if it works and doesn’t seem to lead to any harms, getting stuck with needles may not be such a weird idea after all.

Alice Park is a writer at TIME. Find her on Twitter at @aliceparkny. You can also continue the discussion on TIME’s Facebook page and on Twitter at @TIME.