Can we trust scientists who give TED talks?

I came across an interesting article this morning in Slate questioning recent papers on the “contagiousness” of factors ranging from obesity to divorce. The papers were published in top journals like the New England Journal of Medicine (I wrote this enthusiastic blog post about the findings back in 2008) and have generated a wide range of media attention, including the TED talk which I’ve embedded below.

As far as I know the questions surrounding these papers have been entirely statistical (as opposed to ethical) in nature. Below is the abstract of a critique published in the journal of Statistics, Politics, and Policy earlier this year which nicely outlines the problem of having a high profile paper with a poor stats section:

The chronic widespread misuse of statistics is usually inadvertent, not intentional. We find cautionary examples in a series of recent papers by Christakis and Fowler that advance statistical arguments for the transmission via social networks of various personal characteristics, including obesity, smoking cessation, happiness, and loneliness. Those papers also assert that such influence extends to three degrees of separation in social networks. We shall show that these conclusions do not follow from Christakis and Fowler’s statistical analyses. In fact, their studies even provide some evidence against the existence of such transmission. The errors that we expose arose, in part, because the assumptions behind the statistical procedures used were insufficiently examined, not only by the authors, but also by the reviewers. Our examples are instructive because the practitioners are highly reputed, their results have received enormous popular attention, and the journals that published their studies are among the most respected in the world. An educational bonus emerges from the difficulty we report in getting our critique published. We discuss the relevance of this episode to understanding statistical literacy and the role of scientific review, as well as to reforming statistics education.

I should mention that frankly this stats discussion is well over my head, and it may be that these critiques are thoroughly off base – it took the authors a long time and multiple attempts to get this article published, which could be a sign that there is little weight to the arguments, although it could also be a sign that it’s just hard to get this sort of thing published (our friend Yoni Freedhoff detailed the whole process a few weeks ago, which is where I first heard about these new issues). The point being that these papers are among the most high-profile studies published in my field of research in the past few years, and yet people are now saying things like this:

”[Christakis and Fowler's] errors are in some places so egregious that a critique of their work cannot exist without also calling into question the rigor of review process,”

When I was reading the Slate piece this morning it got me thinking about other recent scientific findings which have been presented in “big idea” forums like TED only to have important questions raised about their veracity.

For example, earlier this year Felisa Wolfe-Simon and other NASA researchers published a paper in Science claiming to have found bacteria which could use arsenic rather than phosphorous as the backbone of its DNA. Shortly thereafter Rosie Redfield wrote a scathing review of the paper on her blog, spawning a massive backlash against the paper in the field as a whole. This backlash prompted Dr Wolfe-Simon and her co-authors to retreat from the media and argue that:

Any discourse will have to be peer-reviewed in the same manner as our paper was, and go through a vetting process so that all discussion is properly moderated.

And yet just a few months later David Dobbs reported that Dr Wolfe-Simon had not only presented her findings at TED, but had also reiterated her paper’s highly disputed conclusions. Here is what David had to say in March:

Apparently the peer-reviewed realm now includes the high-profile TED conference, where on Wednesday Wolfe-Simon talked about her paper. Neither video nor transcript is released as yet [Travis' Note: I still haven't seen them online, but please let me know if someone else has seen them], but accountssuggest she discussed her controversial discovery outside the realm of peer review — in fact, in the most public venue imaginable —and one anonymous source I spoke to today said she repeated the paper’s explicit and disputed claims about arsenic incorporating DNA.

And then there is the case of Marc Hauser, popular author and Harvard researcher who has been under investigation for academic misconduct for the the past year, and whose ultimate fate (as well as his guilt or innocence) remains very unclear. In fairness, he hasn’t presented at TED (although Slate called it “TED-level stuff“), but his popular book Moral Minds certainly places him into the “big idea” category of scientist.

The fact that these eminent “big idea” researchers seem to keep making questionable moral/ethical/academic misjudgments is distressing for a few reasons. First and foremost, it’s because these “big idea” scientists are really the stewards for all of us. TED talks, popular books – these are the way that many non-scientists find out about what is that we do, and why it matters. If the people doing those talks and writing those books turn out to be sketchy then it makes all of us look bad.

But it is also worries me because this is not good for science. I had to stop myself from writing this is not the way that science is done, because that just seems like a cliche in a blog-post mentioning Rosie Redfield and #arsenicDNA. But for science to be done there has to be room for genuine debate, and TED talks don’t seem to have much of that… they seem more like a monologue where you present your ideas as fact. If Dr Wolfe-Simon’s talk had been a debate between herself and one of her critics then I think it would have been far more useful to the advancement of science. And while I realize that advancing science per se is not the purpose of a TED talk, I can’t help but feel that there is something fundamentally wrong about presenting your shiny new finding as fact to a large and very influential audience when it is still eminently unclear whether the finding is legitimate or not. Not that TED talks are wrong, but that it’s dangerous to get too far ahead of the science, or present something as fact when there remain unresolved questions. I have been unable to find a video or transcript of Dr Wolfe-Simon’s talk so I could be off base here, but it certainly seems distressing on the face of it.

The final thing that I find personally distressing about these issues is that I love listening to TED talks and reading books about big ideas. And to be honest I would love to be one of those people who gives those sorts of compelling talks that so clearly demonstrate why an idea or piece or research has meaning outside of the lab. It worries me that other people who share that goal seem to be spreading a message which may not actually represent the “truth”… it makes me nervous about knowledge translation in general if those who are among the most successful are also those pushing the most questionable findings.

These are really good questions you raise. When I was a member of the academic community I was appalled at the contempt some academic scientists held toward those “self promoters” who wrote or spoke for the more popular media outlets. However, sometimes that criticism was not undeserved. Of course I’ve seen the same kind of self promotion in small “selective” scientific communities as well. I think that we cannot paint any group of people as trustworthy or not. We simply have to judge based on the merits (or lack thereof) of the work and actions. What we really need is an educated and thinking society that can see through shoddy data whether it’s presented in a scientific journal or a TED talk or on a blog. That’s the real challenge.

Excellent points. My concern is that (as with the Christakis papers) the science can become so complicated that even specialists have a hard time sussing out the good from the bad. If the reviewers of NEJM aren’t catching a stats problem, then I don’t think lay-people will be able to, no matter how educated. I guess that’s why I’d rather researchers be more clear about which “big ideas” are speculative, and which ones have been established/confirmed by multiple groups.

Excellent point. And I can well imagine someone asking “can we trust scientists who blog?”, and picking examples that reflect poorly on us bloggers as well. That’s why I get so concerned – since there is already skepticism over those scientists who seek to engage the public, I think we need to be careful to make sure that we do so appropriately.

I think one of the big point that we have to stress to the general public is this.

Scientists mess up. Science is dirty, feisty and incorrect, because scientist are dirty, feisty and incorrect.
But Science is good and useful because it tries its best to self correct.

I feel that your preoccupation about these Big Idea presenters is that if they are proven wrong, the public perception of science will be diminished.
I think the solution to that is to make people understand that science is made step by step by correcting the errors that people before us have done. This would dissociate the “correctness” of the Big Idea thinkers from the value that science has overall.
The problem is that most people don’t like things that are uncertain, and we scientist tend to appeal to that by making statements that are more truthful than what they are actually, because nobody likes the “well, the research showed that possibly, with x,y, z caveats, there is a high probability that A is correlated to B. We do not know yet about causation”.
Or even better http://www.phdcomics.com/comics.php?f=1174

But if I have a finding that I know may be questionable (e.g. other people in my field have made legitimate claims that my methods were flawed), is it ethical to go on presenting the data without even mentioning those criticisms. Not that the cases I mention above have necessarily done that, but I think that’s a very different issue from presenting data that we have no reason to question, only to later discover problems with it.

I think as soon as anybody enters the public arena, all bets are off. Over-selling yourself and your work at the cost of accuracy can be central to self-promotion. We see this happening especially in fields like politics and business.

This has put a necessary burden on the community to fact check and do the homework after the talk/speech/claim has been given. Unfortunately, much of the general public doesn’t have time for this, so self-promotion can be dangerous.

It’s nice to see that science is developing a blog watchdog network somewhat similar to those political blogs that fact check public officials.

In the vegan community I see many people jumping aboard the latest guru (Fuhrman, Esselstyn, McDougall etc), book (China Study, Thrive, Skinny Bitch), or movie (Food Inc, Future of Food, Forks Over Knives) bandwagon and thinking it’s the scientific gospel. I try to temper their enthusiasm with caution about BigIdeas and good science but rarely can I communicate it properly.

Are media outlets like books and movies that trumpet BigIdeas helpful to generate interest and therefore further research or does it undermine the public’s understanding even more?

I think a lot of the problem comes in when scientists (or anyone for that matter) forgets that we are human and therefore subject to error. When we cut ourselves off from critical dialog from our peers and from those outside of science who are also thinkers, we get into trouble. When we think that we actually can be totally objective about our own work, we get into trouble. Great (and not-so-great) writers have written about the “entropy” effect of scientists and other professionals isolating themselves from their peers and the public. I think the public presentation of science is terribly important, but it has to be presented as science really is–the incremental learning and advancement of knowledge that is often “one step forward two steps back, oops there was a larger truth we didn’t understand”–which isn’t sexy or instantly gratifying, but it is science. It is a tremendous puzzle, and it is so much fun to scratch your head over it.

Great post and I share your concerns. One of the problems is that Big Idea Science is usually complex and presenting it to the public is difficult because engaging is more important than accuracy. One only has to look at the theory/Theory and belief mix-ups to see why it’s important that everyone understands that science is provisional and define how words are used (that would be a fun talk).

BTW if Christakis did do a TED talk without mentioned the disputes that seems to me to be an attempt to be right by popularizing the idea rather than really examining and proving their hypothesis. Kind of unethical.

Excellent points. I love the TED talks but some of the presentations (e.g. humans evolved from aquatic apes) just stretch the credibility of the presenter. But with a crowd filled with influential people and leaders this could be problematic.

I don’t know what to do about the problem however. TED does a great job of keeping people interested in science which is a good thing. As long as scientists keep other scientists honest it should work itself out.

I think you’re probably right. But it’s still a little disquieting that Dr Wolfe-Simon would go to TED after the backlash her paper generated (and especially after claiming that peer review was the only appropriate venue for discussion of the paper). Whether or not it works itself out (and I think it almost certainly will work itself out), it does seem to suggest that all is not well in science communication world.

This single example is enough to answer the question in your title, and the answer is NO. Mullis has long denied that HIV causes AIDS, he is still listed as sitting on the advisory board of an organization that promotes denial, “Alive and Well.” Both the founder of the organization, Christine Maggiore, and her three year old daughter Eliza Jane, have died of AIDS in recent years (when pregnant Maggiore refused any intervention to reduce the risk of passing HIV to Eliza Jane, an approach her organization advocated for other mothers with HIV). Mullis appears along with his friend Peter Duesberg in a recent fake documentary named “House of Numbers” that was funded by AIDS denialists to promote their false claims. In his asinine interview Mullis offers the “one paper” gambit, in which it is claimed that HIV doesn’t cause AIDS because there is no “one paper” that “proves” it. The interview is on youtube.

In terms of his TED talks, one is entitled: “Kary Mullis’ next-gen cure for killer infections” – try doing a PubMed search to see if you can find any published work by Mullis relating to this purported “next gen cure” (I can’t).

I will quickly pick up your own feed when i can not in locating your e-mail ongoing website link or newsletter company. Accomplish you could have virtually any? Kindly let me know to ensure I might just subscribe. Cheers.

Hello there, just became alert to your blog through Google, and found that it’s really
informative. I’m going to watch out for brussels. I’ll be grateful if you continue this in future.
Lots of people will be benefited from your
writing. Cheers!

What i do not realize is actually how you’re not really much more well-liked than you might now. You are very intelligent. You realize thus significantly relating to this subject, produced me personally consider it from many different angles. Its like women and men are not fascinated unless it’s one thing to accomplish with Lady gaga! Your own great fabrics. Always keep up!

I’ve been exploring for a little bit for any high-quality articles or
weblog posts in this sort of space . Exploring in Yahoo I at
last stumbled upon this website. Studying this info So i’m glad to express that I have an incredibly excellent uncanny feeling I found out
just what I needed. I most surely will make sure to do not put out of your mind this site and
give it a glance regularly.

Thanks a bunch for sharing this with all folks you really recognize what you’re speaking about!
Bookmarked. Kindly additionally visit my web site =). We will
have a hyperlink alternate arrangement among us

Magnificent beat ! I wish to apprentice whilst you amend your website,
how can i subscribe for a blog site? The account helped me a appropriate
deal. I had been tiny bit acquainted of this your broadcast provided brilliant
transparent concept

Subscribe

Mission

Social media provides a tremendous outlet by which to translate and promote scientific knowledge and engage the public discourse. All scientists, researchers, clinicians, government and not-for-profit organizations have much to gain by adopting an effective and viable social media strategy.

Science of Blogging will not only highlight the ways by which social media is changing the way science and research is communicated, but also will provide basic guidelines for those individuals or organizations who seek to use social media to increase the public understanding of scientific research.