Friday, 26 September 2014

Why most scientists don't take Susan Greenfield seriously

Three years ago I wrote an open letter to Susan Greenfield, asking her to please stop claiming there is a link between autism and use of digital media. It’s never pleasant criticizing a colleague, and since my earlier blogpost I’ve held back from further comment, hoping that she might refrain from making claims about autism, and/or that interest in her views would just die down. But now she's back, reiterating the claims in a new book and TV interview, and I can remain silent no longer.

Greenfield featured last week as the subject of a BBC interview in the series Hard Talk. The interviewer, Stephen Sackur, asked her specifically if she really believed her claims that exposure to modern digital media – the internet, video games, social media – were damaging to children’s development. Greenfield stressed that she did: although she herself had not done direct research on the internet/brain impact link, there was ample research to persuade her it was real. Specifically, she stated: “.. in terms of the evidence, anyone is welcome to look at my website, and it’s been up there for the last year. There’s 500 peer-reviewed papers in support of the possible problematic effects.”

A fact-check on the “500 peer-reviewed papers”

So I took a look. The list can be downloaded from here: it’s not exactly a systematic review. I counted 395 distinct items, but only a small proportion are peer-reviewed papers that find evidence of adverse effects from digital technology. There are articles from the Daily Mail and reports by pressure groups. There are some weird things that seem to have found their way onto the list by accident, such as a report on the global tobacco epidemic, and another from Department of Work and Pensions on differences in life expectancy for 20-, 50- and 80-year-olds. I must confess I did not read these cover to cover, but a link with 'mind change' was hard to see. Of the 234 peer-reviewed papers, some are reports on internet trends that contain nothing about adverse consequences, some are straightforward studies of neuroplasticity that don’t feature the internet, and others are of uncertain relevance. Overall, there were 168 papers that were concerned with effects of digital technology on behaviour and 15 concerned with effects on the brain. Furthermore, a wide range of topics was included: internet addiction, Facebook and social relations, violent games and aggression, reading on screens vs books, cyberbullying, ‘brain training’ and benefits for visuospatial skills, effects of multitasking on attention. I could only skim titles and a few abstracts, but I did not come away feeling there was overwhelming evidence of adverse consequences of these new technologies. Rather, papers covered a mix of risks and benefits with varying quality of evidence. There is, for instance, a massive literature on Facebook influences on self-esteem and social networks, but much of it talks of benefits. The better studies also noted the difficulties of inferring causation from correlational data: for instance, it’s possible that an addictive attitude to a computer game is as much a consequence as a cause of problems with everyday life.

Greenfield’s specific contribution to this topic is to link it up with what we know about neuroplasticity, and she has speculated that attentional mechanisms may be disrupted by effects that games have on neurotransmitter levels, that empathy and social relationships can be damaged when computers/games take us away from interacting with people, and that too much focus on a two-dimensional screen may affect perceptual and cognitive development in children. This is all potentially important and a worthy topic for research, but is it reasonable, as she has done, to liken the threat to that posed by climate change? As Stephen Sackur pointed out, the evidence from neuroplasticity would indicate that if the brain changes in response to its environment, then we should be able to reverse an effect by a change in environment. I cannot resist also pointing out that if it is detrimental to perform socially-isolated activities with a two-dimensional surface rather than interacting with real people in a 3D world, then we should be discouraging children from reading books.

Digital media use as a risk factor for autism

My main concern is the topic that motivated me to write to Greenfield in the first place: autism. The arguments I put forward in 2011 still stand: it is simply irresponsible to indulge in scaremongering on the basis of scanty evidence, particularly when the case lacks logical consistency.

In the Hard Talk interview*, Greenfield attempted to clarify her position: “You have to be careful, because what I say is autistic spectrum disorder. That’s not the same as autism.” Yet this is no clarification at all, given that the latest edition of DSM5 states: “Individuals with a well-established DSM-IV diagnosis of autistic disorder, Asperger’s disorder, or pervasive developmental disorder not otherwise specified should be given the diagnosis of autism spectrum disorder (ASD).” Greenfield has had a few years to check her facts, yet seems to be under the impression that ASD is some kind of mild impairment like social gaucheness, quite distinct from a clinically significant condition.

In an interview in the Observer (see here**), Greenfield was challenged by the interviewer, Andrew Anthony, who mentioned my earlier plea to her to stop talking about autism. She replied to say that she was not alone in making the link and that there were published papers making the same case. She recommended that if I wanted to dissent, I should “slug it out” with the authors of those papers. That’s an invitation too good to resist, so I searched the list from her website to find any that mentioned autism. There were four (see reference list below):

We need not linger on the Hertz-Picciotto & Delwiche paper, because it focuses on changes in rates of autism diagnosis and does not mention internet use or screen time. The rise is a topic of considerable interest about which a great deal has been written, and numerous hypotheses have been put forward to explain it. Computer use is not generally seen as a plausible hypothesis because symptoms of ASD are typically evident by 2 years of age, long before children are introduced to computers. (Use of tablets with very young children is increasing, but would not have been a factor for the time period studied, 1990-2006).

The Finkenauer et al paper is a study of internet use, and compulsive internet use by married couples, who were assessed using self-report questionnaires. Frequency of internet use was not related to autistic traits, but compulsive internet use was. The authors did not conclude that internet use causes autistic traits – that would be a bit weird in a sample of adults who grew up before the internet was widespread. Instead, they note that if you have autistic traits, there is an increased likelihood that internet use could become problematic. The paper is cautious in its conclusions and does not support Greenfield’s thesis that the internet is a risk factor for autism. On the contrary, it emphasises the possibility that people who develop an addictive relationship with the internet may differ from others in pre-existing personality traits.

So on to Waldman et al, who consider whether television causes autism. Yes, that’s right, this is not about internet use. It’s about the humble TV. Next thing to note is this is an unpublished report, and not a peer-reviewed paper. So I checked out the authors to see if they had published anything on this, and found an earlier paper with the intriguing title: “Autism Prevalence and Precipitation Rates in California, Oregon, and Washington Counties”. Precipitation? Like, rainfall? Yup! The authors did a regression analysis and concluded that there was a statistically significant association between the amount of rainfall in a specific county, and the frequency of autism diagnoses. They then went on to consider why this might be, and came up with an ingenious explanation: when it is wet, children can’t play outside. So they watch TV. And develop autism.

In the unpublished report, the theme is developed further, by linking rate of precipitation to household subscription to cable TV. The conclusion:

“Our precipitation tests indicate that just under forty percent of autism diagnoses in the three states studied is the result of television watching due to precipitation, while our cable tests indicate that approximately seventeen percent of the growth in autism in California and Pennsylvania during the 1970s and 1980s is due to the growth of cable television.”

One can only breathe a sigh of relief that no peer-reviewed journal appears to have been willing to publish this study.

But wait, there is one more study in the list provided by Greenfield. Will this be the clincher? It's by Maxson McDowell a Jungian therapist who uses case descriptions to formulate a hypothesis that relates autism to “failure to acquire, or retain, the image of the mother’s eyes”. I was initially puzzled at inclusion of this paper, because the published version blames non-maternal childcare rather than computers, but there is an updated version online which does make a kind of link – though again not with the internet: “The image-of-the-eyes hypotheses suggest that this increase [in autism diagnoses] may be due to the increased use, in early infancy, of non-maternal childcare including television and video.” So, no data, just anecdote and speculation designed to make working mothers feel it’s their fault that their child has autism.

Greenfield's research track record

Stephen Sackur asked Greenfield why, if she thought this topic so important, she hadn’t done research on this topic herself. She replied that as a neuroscientist, she couldn't do everything, that research costs money, and that if someone would like to give her some money, she could do such research.

But someone did give her some money. According to this website, in 2005 she received an award of $2 million from the Templeton Foundation to form the Oxford Centre for Science of the Mind which is “dedicated to cutting-edge interdisciplinary work drawing on pharmacology, human anatomy, physiology, neuroscience, theology and philosophy". A description of the research that would be done by the centre can be found here. Most scientists will have experienced failure to achieve all of the goals that they state in their grant proposals – there are numerous factors outside one's control that can mess up the best-laid plans. Nevertheless, the mismatch between what is promised on the website and evidence of achievement through publications is striking, and perhaps explains why further funding has apparently not been forthcoming.

One of the more surprising comments by Greenfield was when Sakur mentioned criticism of her claims by Ben Goldacre. “He’s not a scientist,” she retorted, “he’s a journalist”. Twitter went into a state of confusion, wondering whether this was a deliberate insult or pure ignorance. Goldacre himself tweeted: “My publication rate is not stellar, as a part time early career researcher transferring across from clinical medicine, but I think even my peer reviewed publication rate is better than Professor Greenfield's over the past year.”

This is an interesting point. The media repeatedly describe Greenfield as a “leading neuroscientist”, yet this is not how she is currently perceived among her peer group. In science, you establish your reputation by publishing in the peer-reviewed literature. A Web of Science search for the period 2010-2014 found thirteen papers in peer-reviewed journals authored or co-authored by Greenfield, ten of which reported new empirical data. This is not negligible, but for a five-year period, it is not stellar - and represents a substantial fall-off from her earlier productivity.

But quality is more important than quantity, and maybe, you think, her work is influential in the field. To check that out, I did a Web of Science search for papers published from a UK address between 2005-2014 with topic specified as (Alzheimer* OR Parkinson’s OR neurodegener*) AND brain. (The * is wildcard, so this will capture all words starting this way). I used a 10-year period because citations (a rough measure of how influential the work is) take time to accrue. This yielded over 3,000 articles, which I rank ordered by the number of citations. The first paper authored by Greenfield was 956th in this list: “Non-hydrolytic functions of acetylcholinesterase - The significance of C-terminal peptides”, with 21 citations.

Her reputation appears to be founded on two things: her earlier work, in basic neuroscience in the 1980s and 1990s, which was well-cited, and her high profile as a public figure. Sadly, she seems to now be totally disconnected from mainstream science.

If Greenfield seriously believes in what she is saying, and internet use by children is causing major developmental difficulties, then this is a big deal. So why doesn’t she spend some time at IMFAR, the biggest international conference on autism (and autism spectrum disorder!) that there is? She could try presenting her ideas and see what feedback she gets. Better still, she could listen to other talks, get updated on current research in this area, and talk with people with autism/ASD and their families.

*For a transcript of the Hard Talk interview see here**Thanks for Alan Rew for providing the link to this article

Update: 2nd June 2015: A shortened, version of this blogpost is now posted on the Winnower.

Hertz-Picciotto, I., & Delwiche, L. (2009). The rise in autism and the role of age at diagnosis. Epidemiology, 20(1), 84-90. doi:10.1097/EDE.0b013e3181902d15.

McDowell, M. (2004). Autism, early narcissistic injury and self-organization: a role for the image of the mother's eyes? Journal of Analytical Psychology, 49 (4), 495-519 DOI: 10.1111/j.0021-8774.2004.00481.x

22 comments:

"A Web of Science search for the period 2010-2014 found thirteen papers in peer-reviewed journals authored or co-authored by Greenfield, ten of which reported new empirical data. This is not negligible, but for a five-year period, it is not stellar - and represents a substantial fall-off from her earlier productivity. ... Sadly, she seems to now be totally disconnected from mainstream science."

Hm. It's interesting to see what different benchmarks people use. To me, on the face of it, an average of two papers a year is not "disconnected." Indeed, at many institutions, an average of two papers a year over multiple years would be perfectly respectable, or even superior. So I guess part of the question is what institution she is at, and what their research expectations are.

Thanks for comment, Zen. I would stress that I don't think 2 papers per year should necessarily be seen as an indicator of bad productivity; indeed, emphasis on quantity over quality is a negative force in academia, so 2 meaty papers that have taken a lot of time and thought are preferable to 10 superficial ones. In this case, though, the citation data (and, frankly, also a scan of the abstracts) suggests that quality is not particularly impressive. I'd be surprised if this publication record would have qualified her for inclusion in the REF, for instance (the UK exercise for rating research in higher education).The point I wanted to stress here was how the publication record is at odds with the media's portrayal of Greenfield as "leading neuroscientist". That would suggest her current output should be well above average, with a body of influential papers, and that is just not the case.

I find myself wondering how an analysis of the list of references she provides could be made quicker and easier, particularly for others who don't have your expertise in analysing lists of references. What would be great would be a tool that could take a list like hers and insert a logo next to each one showing whether it was peer reviewed or not, and/or giving the altmetrics of the article. The new tool Pre-Val (http://pre-val.org/) might help with the former, and perhaps some of the altmetrics companies are working on the latter. That would make it easier for the general public to evaluate claims by people like Greenfield who claim there is lots of peer-reviewed work supporting their particular view. I also wonder if there are tools to help compare researchers, which might help address Zen's point. Has Greenfield got an ImpactStory profile I wonder?

While privacy issues don't allow me to confirm it by checking our database, a quick Google for a "Susan Greenfield Impactstory account" yields no results. Though given her sometimes-anti-Internet advocacy, it's not surprising :)

Altmetrics could help others understand the *real* impact (and reception to) the articles she lists, but not in the way you might originally think.

Those articles might be highly shared on social media, giving some the impression that they're of high regard if you were to look at the raw altmetric counts alone.

If we were to use sentiment analysis to show what scientists were *actually saying* about the article, though, that's another story. You'd be able to quickly communicate to others how much rubbish most of the studies are. "Peer-reviewed" badges could be part of that display, communicating to others very quickly whether or not they are *actually* peer-reviewed (rather than jumbled up into a list so long it takes someone as dedicated as Deevy to sort it out).

Impactstory isn't at the sentiment analysis or peer-reviewed badge point yet (nor is any other altmetrics aggregator). But there's more and more talk of things going that way, as the sentiment analysis technology improves. And until then, all the underlying data (who's bookmarking what on Mendeley, what are scientists saying on Twitter vs Facebook, etc) is available in tidy reports.

Anna, thank you for the mention. You have a very interesting idea here. Currently PRE-val can work at the journal/TOC level as well as the article level, but I see no reason why we could not also make it available within reference lists. Full disclosure: I'm the Founder and Managing Director of PRE.

Thanks to Anna and Adam for raising this: I will take a look at Pre-Val.I should add that the figure I give for peer-reviewed papers in Greenfield's list is likely to be an overestimate, as I did not check the status of every journal, rather giving benefit of the doubt to anything that looked plausible.So something to confirm that status of source articles would help. However, there is a bigger issue here, and that concerns the content of articles. I dealt with this issue here (see point 5): http://deevybee.blogspot.co.uk/2011/12/pioneering-treatment-or-quackery-how-to.html. It's a classic tactic of some people to smother their critics with references, creating the impression of a huge weight of evidence in their favour. When you actually get into the reference list, you find many references are of tangential relevance, and some even support the opposing point of view. It's depressing to see Greenfield using this tactic. When I heard she was writing a new book looking at the evidence, I had hoped she would do a proper systematic review. That would have been a really useful addition to the literature.

Perhaps the more important issue about any debate on technology and autism (or beyond) is what effect this might have 'physical' health and wellbeing?

As per the report from Must and colleagues: http://www.ncbi.nlm.nih.gov/pubmed/24113339 sedentary behaviour linked to 'screen time' was more frequent in their autism cohort.

Physical inactivity is a growing issue for quite a few people on the autism spectrum and where this can eventually lead... http://oregonstate.edu/ua/ncs/archives/2014/sep/children-autism-are-more-sedentary-their-peers-new-osu-study-shows

Is this a new problem, though? For most kids, physical activity is often part of the context of social activities -- sports or shared play. Kids on the autism spectrum have had a harder time being included in that for a long time. Granted that this is probably exacerbated in the era of physical activity declining for _all_ kids, but I think this has to also be addressed in the context of benefit kids on the ASD spectrum can get from finding social interactions online.

Autism seems to attract wild hypotheses, learning difficulties surprisingly few. Given the overlap between the two conditions, this is a great pity. As to "leading neuroscientist" this is just TV speak for someone of any notoriety in the Science box: it would be pointless to interview someone who was not "leading" or "top" or "expert". Like the concept of "reference smothering": it carries the clear connotation that the critics have not done the necessary reading (and keeps them quiet for a minute per reference)

Well said Dorothy. I was decidedly unimpressed by her talk at a conference in 2011, which I have finally got around to describing today. I have put up the pdf of her powerpoint, and checked a couple of the sources cited too, with similar results to your own check - the only good sources don't support her arguments and are thus tangential (and one might argue, present only to give a superficial impression of hard science). Its here if you want to take a look: http://wp.me/p53qtA-u The blog is called "High on scare, low on science: a tale of charity, politics and dodgy neuroscience" and I'd love to hear your feedback, as blogging is a relatively new thing for me!

As someone with an interest in scientific evaluation, your blog was quite interesting. Thanks. However, I was let down by your uncritical acceptance of literature citation as an indicator of anything, particularly influence or significance. Sorry, but I don't have time to explain why but I'd encourage you to consider why people cite a paper (sometimes to critique or discount it) and how a paper can influence yet not receive a citation (e.g., in decision making.)

I meant to comment earlier, but I want to thank you for taking the time to so very carefully and methodically debunk these claims, as well as describing how you've gone about it. Those who are convinced by evidence as opposed to bluster and sales(wo)manship are certain to think your argument the far more convincing.

"I cannot resist also pointing out that if it is detrimental to perform socially-isolated activities with a two-dimensional surface rather than interacting with real people in a 3D world, then we should be discouraging children from reading books."Well, i also cannot resist pointing out that it's ridiculous to compare a passive page with nothing but words to an active screen with moving images for "interactivity".

Gamers HeavenWe at Gamers Heaven love gaming and so we dedicate our work in making your playing experience as enjoyable as it can be. Our aim for all our products is one thing, helping you have the best gaming experience possible.Buy Chair: http://gamersheaven.co.uk/