Brain training app brings big data to human cognition

Checking the impact of age and alcohol with a sample size of 6 million.

One of the problems with cognitive and behavioral research is getting a good cross-section of the general population. Although they're convenient to work with, a couple hundred college students rarely represent the full diversity of human capability and behavior, yet that's exactly what many studies rely on. But a brain-training game may now provide access to data on scales that behavioral scientists probably never dreamed of. With a user base of over 35 million, the data obtained through the game could help us tease out very subtle effects. But as a start, a team of researchers have focused on some simpler questions: how aging and alcohol affect our ability to learn.

The software is less a game itself than a game and survey platform. Developed by a company called Lumosity, it's available on mobile platforms and through a Web interface. The platform can run a variety of games (a typical one asks users to answer math questions that appear in raindrops before they hit the ground), all with an emphasis on brain training. A few games are available for free and users can pay to get access to more advanced ones.

The scientific literature on brain training games is a bit mixed, and there's some controversy about whether the games improve mental function in general, or only those specific areas of cognition that the game focuses on. Lumosity clearly argues for the former and one of its employees pointed Ars to a number of studies that he felt validate the company's approach. What's not in doubt, however, is that it has a huge user base with over 40 million registered users. And because the Lumosity platform is flexible, it has been able to get basic demographic information from many of those users; they and others have also filled out personality profiles and other assessments.

Finally, people often play the same game multiple times, potentially over several years. This allows researchers to track both how the game affects the players' abilities to process certain information (namely, do they get better at it?) and may eventually allow some longitudinal studies of a population over time.

For now, however, the company (along with a couple of researchers at outside institutions) is simply trying to validate that its dataset gives it the sort of results that have been hinted at by other work. And two of the things Lumosity started with were sleep and booze.

A lot of the Lumosity client base has answered survey questions about how much sleep they get and how often they have a drink, so the authors correlated that with their scores on a number of tests, targeting working and spatial memory as well as simple arithmetic skills. The key thing in the work was the number of subjects: over 160,000 in two cases and 125,000 in the third.

In all tests, performance sharply increased as people got more sleep, peaking at seven hours of sleep a day. From there, things showed a more gradual decline, with people who got 10 hours of sleep scoring about the same as those who got five. A similar pattern, with a much lower threshold, was apparent when it came to alcohol intake. People who had one or two drinks a day outperformed the teetotalers, but anything above the two-drink threshold saw a drop. In order to see scores similar to those who abstained, people had to have four or five drinks a day.

From there, the researchers turned to learning by tracking people's performance on a single game over 25 iterations. In general, baseline performance dropped with increasing age. But there were distinct differences between the different types of tasks. For example, spatial and working memory seemed to decline steadily from the 20s on. Math skills decayed more slowly and didn't start until the 30s, while verbal fluency stayed stable into the 50s. This seems to largely be a result of people's ability to learn. Improvements in verbal and arithmetic skills stayed largely flat across the range from 20 to 70 years, while the two memory tests showed declines in improvements across roughly the same time.

Overall, the authors admit that the results don't tell them much about the mechanisms behind the effects they're seeing (for example, they note "the apparent cognitive advantage for those who report moderate alcohol intake may be in part due to increased social and cognitive engagement compared to those who report little or no alcohol consumption"). But it is clear that the data available via Lumosity should let people test various mechanisms on a scale that would be impossible without dragging the entire student population of several universities into a behavior lab.

To aid in this, the company has set up a group that evaluates outside research proposals and provides access to the data to any that meet its criteria. Once access to the data is granted, Lumosity takes no further part, allowing researchers to evaluate the data independently. The authors of the current paper were largely Lumosity employees, but it's possible that this will eventually be a relative rarity.

I just looked at it on flickr and the picture at the top of this page is only allowed to be used noncomercially. Is it's use an accident, or what's the justification?

Editorial use isn't the same as commercial use. As opposed to say, using the image in a banner ad. You can still be a for-profit editorial enterprise. So the use isn't an accident, we definitely pay attention to the terms of the CC licensing (such as the restriction on modifying for instance). Our policy is also to pull any image if the rights holder objects to our use, regardless of what lawyers say is allowed.

Reading Ars science articles works pretty much the same way to me. You have spent hours reading the articles and spent the next month trying to figure out what it says. Don't you think in a way it is a good exercise to our brains? I think so. My brain seems to be more sensitive than ever I have. And here you have often read someone commenting he's pulling his hairs out for not understanding a word the article is saying? And how often have you passed out right in the mid of reading Dr. Trimmer's article due to overloading your brain's memory? Things like that happens. :-)

I wonder if there might be a spin available on mining the NSA-scale meta-data for for use in epidemiology?

Some might even consider "opting-in" to allow personally identifiable correlation for studies aimed toward the greater public good, or allow opt-in to be screened by insurance providers for discounts or associating with risk pools.

Editorial use isn't the same as commercial use. As opposed to say, using the image in a banner ad. You can still be a for-profit editorial enterprise. So the use isn't an accident, we definitely pay attention to the terms of the CC licensing (such as the restriction on modifying for instance). Our policy is also to pull any image if the rights holder objects to our use, regardless of what lawyers say is allowed.

That got me even more curious, so I went and looked at the CC legalese. The relevant bit seems to be section 4c: "You may not exercise any of the rights granted to You in Section 3 above in any manner that is primarily intended for or directed toward commercial advantage or private monetary compensation."

Looks good to me. To my non-lawyer eyes, a jello brain photo on an article about brain research seems unlikely to sell more ad clicks or impressions, so it's primarily for snarky humor, not commercial advantage.

What's the best way to increase your score in an IQ test? Practise taking IQ tests. When IQ tests became big business 40 years ago a whole industry developed around teaching people how to fake their way to a higher score. It didn't make them any smarter.

'As lead researcher Adrian Owen says: "You're not going to get better at playing the trumpet by practising the violin."'

What's the best way to increase your score in an IQ test? Practise taking IQ tests. When IQ tests became big business 40 years ago a whole industry developed around teaching people how to fake their way to a higher score. It didn't make them any smarter.

'As lead researcher Adrian Owen says: "You're not going to get better at playing the trumpet by practising the violin."'

Except that you will, at least in some ways. People who are already trained in one instrument will generally pick up another instrument much more quickly than people who have no musical training whatsoever. You don't get better at trumpet fingering by practicing violin, but your ability to read music, detect pitches and tuning, and things like that will get better and that will improve your trumpet-playing.

Also the moderate-alcohol-correlates-to-better-scores thing seems like it's more likely a result of income/alcohol consumption correlations than to any underlying effect of alcohol on cognition (people who consume in the 1-2 drinks-a-day range tend to be middle- and upper-income people who are better-educated and healthier than average). I'd be willing to bet if you controlled for income, education, and age (people's alcohol consumption tends to fall as they age) these correlations would disappear.

Anyway, there seems to be dependence between the variables. E.g, the max score for hours of sleep per day is 24, but on the same task for drinks per day it is 24.5. Since the number of subjects is really large, they cannot be independent.

And what's weird is that performance on Speed Match is optimal at 2 drinks per day, while both other tasks are optimal at 1. So there's clearly not a single effect.

And that the standard deviation (I guess that's what the bars are reporting, but who knows? Might be quartiles, there's no description) for hours of sleep are much smaller than those for drinks per day is also remarkable. And that's not just the number of subjects, since there can't be that many subjects that sleep less than 4 hours per day or more than 10. The effects from sleep are also larger than from drinks.

As if we needed more proof to Ben Franklin's oft quoted saying, "Beer is proof that God loves us and wants us to be happy."

While I wish it were true, Ben Franklin never actually said that. I love beer, I probably love beer more than most people, but it still annoys me seeing that quote every time I go to my local homebrew shop.

Also the moderate-alcohol-correlates-to-better-scores thing seems like it's more likely a result of income/alcohol consumption correlations than to any underlying effect of alcohol on cognition (people who consume in the 1-2 drinks-a-day range tend to be middle- and upper-income people who are better-educated and healthier than average). I'd be willing to bet if you controlled for income, education, and age (people's alcohol consumption tends to fall as they age) these correlations would disappear.

I would think the cohort of people who log into Lumosity.com and play brain-training games on the internet is already biased towards people of higher education and income. I also don't really see the correlation between alcohol consumption and income or education level. They sell plenty of booze to homeless people.

Apparently this was a science news article, not a software article. The link in the article goes to a report of a journal abstract.

It's very cool to know that programs/apps can be used to provide entertainment/educational value to a consumer, yet also be a test subject as part of a scientific research project. Data is collected all the time, but we tend to think of it only in terms of statistics for marketing and business. It's refreshing to be something more than a source of revenue to be exploited.

It also seems like a great idea in terms of research design and methodology for recruiting test subjects, application of tests, and tracking of subjects - possibly long-term as it would facilitate getting those subjects to return for repeats and further tests. It also clearly has potential in terms of studies at least being able to be partly self-funded. While I'm not sure that I would want to pay to be a participant in someone's study, registration/payment methods would likely help with collecting/ensuring some degree of accuracy for collecting test subjects' personal data, which is probably one of the biggest issues with online research.

What I find... interesting, or mildly disturbing, is that when you do go to the Lumosity website, the ToS and Privacy Policy clearly indicate that data is being collected for business purposes. Clearly, it does indicate that there is some research, however it leads the reader to assume that the data is only for improving their service/product.

Quote:

http://legal.lumosity.com/privacy_policy

We also use the information to manage and improve our Site, our games and our training. We may use your information, in aggregate form and in collaboration with outside researchers, to measure and quantify the effectiveness of our program.

There is no direct disclosure that the "program" is (part of) a collaborative research program they call the Human Cognition Project.If you were to simply join for the "brain training," read the ToS and Privacy Policy, without clicking on the link for the Human Cognition Project (which doesn't appear to be of any immediate importance to site/app users), you would have no clue that you were part of an academic neuroscience/psychology of learning study.

I would think the cohort of people who log into Lumosity.com and play brain-training games on the internet is already biased towards people of higher education and income. I also don't really see the correlation between alcohol consumption and income or education level. They sell plenty of booze to homeless people.

You don't get it, do you? Plenty of booze to homeless is a correlation between alcohol consumption and income. Young people cannot drink and have a low income, another effect. There are plenty of other effects imaginable, e.g. that a group of people who cannot drink because of medical condition, medication, being too young or too old, whatever, is dragging the score for 0 drinks per day down. While it's a funny article, it's totally useless for drawing any kind of conclusion.

I would think the cohort of people who log into Lumosity.com and play brain-training games on the internet is already biased towards people of higher education and income. I also don't really see the correlation between alcohol consumption and income or education level. They sell plenty of booze to homeless people.

You don't get it, do you? Plenty of booze to homeless is a correlation between alcohol consumption and income. Young people cannot drink and have a low income, another effect. There are plenty of other effects imaginable, e.g. that a group of people who cannot drink because of medical condition, medication, being too young or too old, whatever, is dragging the score for 0 drinks per day down. While it's a funny article, it's totally useless for drawing any kind of conclusion.

Perhaps I should have explicitly stated positive correlation, but I didn't realize I was going to get a reply from a patronizing pedant. Or maybe you just didn't read the quoted portion to which I was responding.

I just looked at it on flickr and the picture at the top of this page is only allowed to be used noncomercially. Is it's use an accident, or what's the justification?

Editorial use isn't the same as commercial use. As opposed to say, using the image in a banner ad. You can still be a for-profit editorial enterprise. So the use isn't an accident, we definitely pay attention to the terms of the CC licensing (such as the restriction on modifying for instance). Our policy is also to pull any image if the rights holder objects to our use, regardless of what lawyers say is allowed.

I see that there is much that I don't understand, thanks for the followup!

What's the best way to increase your score in an IQ test? Practise taking IQ tests. When IQ tests became big business 40 years ago a whole industry developed around teaching people how to fake their way to a higher score. It didn't make them any smarter.

'As lead researcher Adrian Owen says: "You're not going to get better at playing the trumpet by practising the violin."'

Except that you will, at least in some ways. People who are already trained in one instrument will generally pick up another instrument much more quickly than people who have no musical training whatsoever. You don't get better at trumpet fingering by practicing violin, but your ability to read music, detect pitches and tuning, and things like that will get better and that will improve your trumpet-playing.

Not so fast. You're saying that practising reading music makes you better at it, whether that practice is part of trumpet or violin rehearsal. This is hardly surprising, you're getting better by practising the skill you want to get better at. It doesn't contradict the fundamental point that expertise in these skills is non-transferable. Practising reading music doesn't make you any better at reading Chinese.

Sure, some skills (such as reading) are tools which are essential for learning other skills. But this is only true for a limited set of abilities. Please explain how learning to spot a bird flashed briefly on a screen is going to be useful in learning something else, It's not, unless that secondary skill depends on being able to remember the position of briefly-glimpsed objects.

All I can say is that anyone gullible enough to pay £9 a month to do a few basic perception/memory tasks every day certainly needs some kind of help....

Perhaps I should have explicitly stated positive correlation, but I didn't realize I was going to get a reply from a patronizing pedant. Or maybe you just didn't read the quoted portion to which I was responding.

If there is any, it would be negative, I guess (higher income, less alcohol), with the distinct possibility of a relation between the two that's not a straight line, but more along the line of the performance, which is what the statement you quote hints at. Your statement doesn't contradict it, although your tone suggests you think so: you've got no evidence for it. Using condescending words doesn't improve that.