Middle aged men who drink 2.5 drinks per day may accelerate memory loss by six years, according to a new study.

Researchers from University College London in the UK assessed the drinking habits of 5,054 men and 2,099 women at three different times over a ten year period. When the participants turned 56, they took the first of three tests of their memory and executive function over the next 10 years.

They report in the journal Neurology that men who downed an average of 2.5 drinks per day showed signs of memory loss sooner than men who who didn’t drink or men who were lighter to moderate drinkers. Even after they controlled for memory-affecting factors such as their diet and exercise habits and occupation, the connection held. The researchers didn’t find a similar trend among women, although the heavier drinking women did show deficits in organization and planning skills.

A drink was classified as beer, wine or liquor, and while those imbibing liquors like vodka, gin or whiskey showed the fastest declines, there didn’t appear to be any differences in memory loss among those drinking beer or wine.

It’s not the first study to document the negative effect that drinking can have on cognitive functions. But it is among the first to look at its effects starting in younger, middle aged people. And it demonstrated how little alcohol it takes to affect higher order functions like memory. According to the National Institute on Alcohol Abuse and Alcoholism, for men, drinking up to four drinks a day is considered low risk drinking, while up to three drinks daily is considered low risk for women. When measuring potential health risks associated with drinking, the Centers for Disease Control says moderate consumption involves up to two drinks daily for men and one alcoholic beverage a day for women.

The researchers speculate that alcohol somehow interferes with blood flow to the brain. Previous studies showed that excessive drinking can also damage nerve cells that can affect the brain’s cognitive functions, but these latest findings suggest that the harm may begin sooner than experts had thought – and last longer than they had expected as well.