TED organizers help sniff out bad science for TEDx

New guidelines help local events identify pseudoscience then keep it off stage.

The organization behind the TED talks took what might have been an embarrassing bit of publicity and turned it into an opportunity to define what constitutes decent scientific information. Although TED is most famous for organizing two sets of talks each year, it has branched out into licensing local affiliates that host smaller events. Many of these pick up on the main program's themes of tracking developments in science, technology, and medicine.

Unfortunately, the TEDx events don't always have access to the same competencies TED proper does. This has sporadically resulted in problems with speakers too far outside the mainstream. Take a case in Valencia, Spain for example, where things like crystal therapy and homeopathy ended up being promoted by speakers. That, in turn, led to a Reddit thread highlighting the problem.

Rather than pretending it never happened, TED's editorial staff got together and put down some guidelines for TEDx organizers. The new guidelines don't draw a sharp line between science and everything else (partly because there's no consensus on how to do so), but they do highlight a lot of the warning signs to determine if a potential speaker's focus might not be on solid ground.

Emily McManus, one of the TED editors who helped to put the guidelines together, told Ars "this letter wasn't about TED laying the science smackdown on TEDx," but rather an attempt to "give the TEDx community some tools to make better decisions when they curate." ("Curate" in this case meaning arrange speaker lineups.) She also pointed to a Reddit post by the director of TEDx, Lara Stein. Stein wrote that, in the past, issues like this had normally been handled informally. With the increased profile, the editorial team felt like it was time to make a more public statement.

The guidelines themselves are pretty heavily focused on fringe medicine, with an emphasis on recognizing when "cures" haven't been tested or identifying when speakers have a financial interest in promoting their ideas. Beyond that, there is advice on recognizing behavior that would get someone a high score on the crackpot index, like disdain for the scientific establishment and claims of persecution. All of it is packaged into language that's casual and sometimes humorous. "It would have been weird to recast our final draft into a lot of 'thou shalt nots,'" McManus told Ars. "Talking down to people or yelling at them just makes them retreat."

Other aspects of the advice should be on anyone's check list when faced with a scientific claim: look at the credentials, affiliations, and research history of the person making the claims. Then, if possible, talk to an independent expert in the field to see if there's any consensus about the ideas in question.

What isn't there is a checklist that will leave you knowing whether something is or isn't science. "There's less of a clear line than you might wish," McManus said. (And she's right; philosophers have been arguing about this for decades.) Still, she felt it should be possible for even non-experts to tell whether a given claim has been rigorously tested.

This also doesn't mean that either TED or TEDx should be expected to go strictly mainstream. McManus said the organization won't shy away from presenting "iconoclasts and controversial ideas;" what it will do is try to provide enough context so anyone who sees the talk knows if ideas are on the fringes. An example of this we discussed is a talk by a proponent of the "aquatic ape" idea. This proposal, which posits that humans evolved in an aquatic environment, is based on a very selective reading of the available evidence. It's completely incompatible with the fossil record.

To McManus, there's enough context in the aquatic ape talk that a viewer should be able to know that it's a fringe idea where the evidence is lacking. "The issue for me is when someone takes the stage and presents non-scientific work in a vacuum such that the average viewer doesn't know whether it's true or not, or runs down the scientific establishment to erode trust in it for their own cynical purposes," she told Ars.

So, are the guidelines any good? In general, I think they are, although they're a bit narrowly focused. A lot of the behaviors they note as signs of potential problems are ones I use myself. And, although their list of "red flag topics" (which includes GMO food and autism) is much shorter than mine, it's definitely a subset of the list I use. It correctly notes these are just warning signs, meant to indicate you've got a topic or person that needs to be approached with caution then vetted through outside expertise.

McManus emphasized the guidelines are meant to be the first shot at an evolving set of advice specifically targeted to TEDx organizers. But, at least in their first incarnation, they seem to be worth the consideration of a wider audience.

72 Reader Comments

Wouldn't it be better to implement some sort of peer review system, by which experts in the field can oust a speaker from the schedule? I'm not understanding why lay people should be making the schedule at all, or at least without being forced to consult a few experts.

I tend to avoid most TEDx talks (I have checked out local TEDx listings). My friend's ex girlfriend gave one, and it was mostly just opinions based on personal experiences at the end of which I felt glossed over the entire subject matter on which she gave the talk on. And I had a chance to personally meet Jake Shimabukuro who did TEDx Japan, but I was too hungover the day I was supposed to meet him - suffice to say I've been close to some people who have been involved in TEDx.For me the problem with TEDx, and I'm glad that they're laying the groundwork, was exactly this, there was very little peer review around the talks, and in part that was fuelled by TEDx going mainstream and the whole psuedo-intellectual status quo of saying you've attended one. There have been some amazing speakers IIRC for some of the local TEDx talks, but those get diluted by the other speakers whom without some peer review, is questionable. Like my friend's ex-girlfriend and the talk she gave. Total WTF moment.

Wouldn't it be better to implement some sort of peer review system, by which experts in the field can oust a speaker from the schedule? I'm not understanding why lay people should be making the schedule at all, or at least without being forced to consult a few experts.

Because TEDx events can be very small. 50 attendants isn't unheard of for a TEDx organization's first event. Good luck trying to haul on board enough experts that every issue has a diverse set of experts ready to curate. It's also worth noting that the TED brand has a pretty low profile in many countries, so it's not a given that the local university faculty will jump at the chance to participate.

OTOH, you could just take the approach that when esteemed physicists stray outside their realm of expertise they are actually more likely to make stupid statements than the man on the Clapham Omnibus - see Roger Penrose and microtubules...

TEDx is largely bad. Some of the earlier TEDx events were nice, which were better funded and focused on certain fields (e.g. A TEDx on government and education). But now a good TEDx talk is a needle in a very large haystack.

I think TED needs to divorce itself from TEDx. Continue providing the information and assistance they currently provide. Be far far more selective about which talks make it to the website (don't add the talks by default). And change the TEDx name, to discontinue the association with TED.

Perhaps a solution would be for the organizers to publish a handout detailing the peer review and hard scientific evidence (if any) behind each speaker's talking points.

That way you don't exclude those whose ideas haven't been proven, but it's made clear where there is actual evidence backing something up and where something is purely theoretical and possibly a load of malarkey.

I find some of the pseudo-science stuff entertaining. Maybe TEDx isn't the place for it, or maybe it could be as long as attendees were made aware of what is and is not provable using standard scientific practices.

Presumably one of the guidelines is "if your talk is about a paper that has since been withdrawn, then no". Although there is at least one Ars Technica reader who still seems to believe that vaccinations cause autism.

The benchmark should really be: peer reviewed, properly designed, reproducable and double-blinded, has not been flat out contradicted by other evidence, the speaker has appropriate qualifications in the subject about which they speak, those qualifications are not from a quackiversity.

Sounds like a bunch of bullshit to me. I think the most important thing is the person's motives. besides that Either you learn something from the talk or you dont. Then you go and do some research for yourself and/or apply it to your life and see if it works (if possible) If not you havent lost anything because everything is a learning experience. Scientific observations help, but there not the end all be all to all to vetting every idea.

You make it sound so easy and cut-and-dry. Why, as just one example, hasn't the whole "vaccines cause autisim" thing sorted itself out yet then?

edit: among people who decide, on an individual basis, whether to vaccinate or not. I think this matters.

I'd be more impressed if the main TED talks actually followed these guidelines. I remember a talk by a woman who's degrees and career were all in money management, but her talk was on linking increased allergies in kids to modern Ag practices. As some one who IS qualified to judge, I can assure you her talk scored pretty high on my BS meter. At that point I stopped considering TED a source of reliable science.

Some of their talks are still good, as in entertaining. I loved Mike Rowe's talk (from Dirty Jobs), but they are not science presentations IMO.

What isn't there is a checklist that will leave you knowing whether something is or isn't science. "There's less of a clear line than you might wish," McManus said. (And she's right; philosophers have been arguing about this for decades.

I don't understand why this should be so hard. The key question is always "do they tell me how I can find out of their claims are true for myself?". This basically amounts to replicability. If they don't it is not science.

Of course scientists also INTERPRET findings, and in doing so they have to speculate somewhat. However when they do so they explain how to determine if their speculations are correct or not. This is of course reflective of the same basic principle, i.e. if you can't tell me what the world would be like if your claim was true your claim is neither true nor untrue but quite literally meaningless.

This basic principle goes back at least as far as Compte, and is universally accepted in all the sciences. Indeed TED acknowledge it very clearly in the first of their guidelines: "Marks of good science: It makes claims that can be tested and verified".

I'm a little disappointed that Mr. Timmer should present something so simple and so universally accepted as somehow uncertain and arcane.

I'd be more impressed if the main TED talks actually followed these guidelines. I remember a talk by a woman who's degrees and career were all in money management, but her talk was on linking increased allergies in kids to modern Ag practices. As some one who IS qualified to judge, I can assure you her talk scored pretty high on my BS meter. At that point I stopped considering TED a source of reliable science.

I always thought the TED presenters were shooting from the hip as well. Then again I thought that was the whole point of TED, observation versus rigor. Insight versus science.

That reddit thread singles out speakers like Amma Amritanandamayi Devi. Are they going to ban religious and humanitarian speakers from non-christian cultures, since they are not science?

Only if they claim their religious beliefs can magically influence the real world (faith healing for example), and the same should apply for stuff from Christian cultures. Philosophy is fine, it's when they claim to have measurable claims which have no evidence backing them

re: telepathy (that term though is vague and baggage laden), this matter has actually been proven (take a look at the work of Russel Targ and physicist Harold Puthoff), though the mechanism is still unknown.

I volunteered to help at a local TEDx. It was profoundly disillusioning. I was on the refreshment committee, and they would only serve water and various infusions of wheat grass and likewise. All food had to be vegetarian, vegan when possible. I suggested that nobody ever seemed to mind cold cuts and a coke at any gathering I'd ever been to, but they explained that they would only promote non-chemical blah blah blah. They were polite and all, but I had to quit. I can't stand that kind of errant nonsense.

Has anybody suggested using Palantir, a link analysis tool favored by troops in Afghanistan for detecting IEDs, and used in managing relief resources after Hurricane Sandy? A tool like that would replace the subjective with a more objective approach to determining what constitutes "bad science".

I find some of the pseudo-science stuff entertaining. Maybe TEDx isn't the place for it, or maybe it could be as long as attendees were made aware of what is and is not provable using standard scientific practices.

Psuedoscience loves people who are more interested in being entertained than informed, and certainly need the veneer of respectability TED gives them. They also love the ability to spout "expertise" without actually needing to back their data, which TED also gives them in spades, apparently. The brand suffers, deservedly from the focus on pop science and "interesting" over factual.

I'd rather TEDx didn't exist than have people who provide faith healing over the phone be given it as an outlet and validation.

Yeah, defenses like that indicate that the brand deserves a nice quiet smothering, not our sympathy. It drags all other talks down to match the factless blatherings. Blame the lack of rigor, not those who want to keep things reality-based.

I'd be more impressed if the main TED talks actually followed these guidelines. I remember a talk by a woman who's degrees and career were all in money management, but her talk was on linking increased allergies in kids to modern Ag practices. As some one who IS qualified to judge, I can assure you her talk scored pretty high on my BS meter. At that point I stopped considering TED a source of reliable science.

Some of their talks are still good, as in entertaining. I loved Mike Rowe's talk (from Dirty Jobs), but they are not science presentations IMO.

You should be challenging TED on talks like this when the speaker is not a subject matter expert.

They were polite and all, but I had to quit. I can't stand that kind of errant nonsense.

That's why I gave up on organized skepticism and atheism after dipping my toes in a few years back. These people would scrutinize all manner of science woo, but when it came to politics the most ridiculous nonsense regularly got a pass. Just too many people who got rid of religion and replaced it with equally moonbatty ideology. Meh...

That's why I gave up on organized skepticism and atheism after dipping my toes in a few years back. These people would scrutinize all manner of science woo, but when it came to politics the most ridiculous nonsense regularly got a pass. Just too many people who got rid of religion and replaced it with equally moonbatty ideology. Meh...

While I agree with the prevalence of MRAs/gender purists and other pseudoskeptics in the "organized" movement, shrugging with a false equivalency is pretty defeatist.

re: telepathy (that term though is vague and baggage laden), this matter has actually been proven (take a look at the work of Russel Targ and physicist Harold Puthoff), though the mechanism is still unknown.

Tell me how to not be defeatist in this world. I'm about out of inspiration on that front.

I didn't quit atheism or skepticism, if that's what you meant. I just don't partake in the organized bits- the ones that advertise. I'm not even sure what a gender purist is. Google returned a paltry number of hits leading to baffling things.

re: telepathy (that term though is vague and baggage laden), this matter has actually been proven (take a look at the work of Russel Targ and physicist Harold Puthoff), though the mechanism is still unknown.

Well, it obviously transmits thoughts and images via the aether.

I'm off to a bloodletting at the barber.

You seem to prefer snarky replies to buttress your opinions over actually looking at the research.

The US military developed a trainable, repeatable protocol called Remote Viewing. The training documents are freely available (declassified through FOIA), though >95% of the materials in that program remain classified. An independent commission hired by the CIA even found it to be real, that report too is available online. There are civilian training classes, or anyone can read the FOIA docs to learn the protocol. I went through formal training, and it absolutely works. There is a strong cultural aversion to it, so I'm not surprised I got a hostile response.

You seem to prefer snarky replies to buttress your opinions over actually looked at the research...I went through formal training, and it absolutely works. There is a strong cultural aversion to it, so I'm not surprised I got a hostile response.

The fact that you originally wrote "proven" instead of "suggested" demonstrated that you are unfamiliar with the scientific method and are not in a position to evaluate the studies. Now you write "absolutely," which is another scientific no-no.

I'm familiar with remote sensing and the very, very shaky methodology used for it. It seems, however, that you're not familiar with science and skepticism. Extraordinary claims require extraordinary evidence. Remote sensing has something that sometimes, under the right conditions, shown to the right people, with the right bias, resembles evidence.

You seem to prefer snarky replies to buttress your opinions over actually looked at the research...I went through formal training, and it absolutely works. There is a strong cultural aversion to it, so I'm not surprised I got a hostile response.

The fact that you originally wrote "proven" instead of "suggested" demonstrated that you are unfamiliar with the scientific method and are not in a position to evaluate the studies. Now you write "absolutely," which is another scientific no-no.

I'm familiar with remote sensing and the very, very shaky methodology used for it. It seems, however, that you're not familiar with science and skepticism. Extraordinary claims require extraordinary evidence. Remote sensing has something that sometimes, under the right conditions, shown to the right people, with the right bias, resembles evidence.

Is snark a scientific no-no?

Well, it's not "remote sensing", it's Remote Viewing, and Remote Viewing is a U.S. military derived protocol. You got a link to the "very shaky methodology" you refer to?

The "extraordinary evidence" exists too, you just haven't checked it out. Like I said, you can learn it yourself and see for yourself... no need to even take my or anyone else's word for it.

If you were familiar with it you wouldn't call it remote sensing. It's Remote Viewing, and it is a military derived protocol.

Which is synonymous with remote sensing.

Quote:

I stand by "proven", it is not "suggested", it is proven (though whether I use the word proven or absolutely does not alter whether or not it exists).

The germ theory of disease is very, very, very strongly suggested. Science does not prove, for that would mean finality. Instead, the scientific method seeks to falsify ideas, all of which are tentative, in an effort to disregard ideas that don't hold up under new evidence. It's geared around the idea that humans make mistakes, sometimes systematically, so whatever we've found might need to be corrected down the road. In fact, it usually does.

Quote:

Is snark a scientific no-no?

Not that I've heard.

Quote:

The fact that you and others look the other way says more about you than the proof.What does it say? That I'm "culturally biased?"

Quote:

The "extraordinary evidence" exists too, you just haven't checked it out.

Yeah, I did. It's just that I'm skeptical. The "evidence" isn't actually evidence.

Quote:

Like I said, you can learn it yourself and see for yourself... no need to even take my or anyone else's word for it.

For someone with so much evidence, you sure are vague. You should be able to find me some peer-reviewed evidence with solid methodology. I'm waiting. And no, "I've done it, and it works" doesn't count.

"Vague" is not the right word... I'm being standoffish... standoffish because I'm not sure I want to have a conversation with someone who thinks snark makes a good introduction. Be nice and we can talk, be snarky and I will be done.

From a "pure science" perspective, the best collection of scientific studies is in the book "Distant Mental Influence".

No, it's not free. Yes I own a copy and have read it. It has dozens of studies with solid methodologies:

RV was as much a military endeavor as it was a scientific one. It was created by Stanford Research Institute for the US Military, and most of it remains classified. Because of its classified nature, they didn't farm it out for 'peer review'. That said, the RV manuals have been declassified, and there are some sample sessions there:

Keep in mind when reading this that AIR did not have access to any classified material at the time that they wrote their report, and at that time it was 98% of the material. AIR based its conclusions on the 2% that the government was willing to declassify (that pretty much invalidates their negative findings). So it is no surprise nor is it an indictment of the program that AIR concluded that there was insufficient evidence of actionable intelligence to be gained. What should be surprising to you is that they found "a statistically significant effect has been observed in the laboratory".

Less scientific but no less valid, read the solving of a Denver murder case (page 4):

Snark hurts the spirit of inquiry... let's have a friendly conversation instead.

I appreciate your honesty in posting entirely unrelated "sent" and "received" images, but do you think that they are supposed to be convincing to those not already credulous in the matter?

A close relative of mine was a part of http://en.wikipedia.org/wiki/Project_Grudge . They claimed that they had "reliable" evidence of US and Soviet remote viewing successes, but yet post-retirement from military intelligence, befriended a number of cranks and scammers. If that technology existed, it would be repeatable, verifiable, and not limited to the self-deluded and those trying to exploit others (like, for example, those who claim to have "solved" any cases when the reality is far more banal.)

Tell me how to not be defeatist in this world. I'm about out of inspiration on that front.

I didn't quit atheism or skepticism, if that's what you meant. I just don't partake in the organized bits- the ones that advertise. I'm not even sure what a gender purist is. Google returned a paltry number of hits leading to baffling things.

Ah, I wish I could. Avoiding the "atheist movement" and those who still stick by Dawkins' words after his "Dear Muslima" rant is a good start.

"I'm not even sure what a gender purist is"

Mostly irrelevant to the matter, but I was referring to those who believe that roles are essential, not arbitrary, and that regressing to the mythical 1950s (by rolling back the gains of feminism) are all essential. Essentially the cave-troll pseudoskeptics that harp on women, blaming them for firm gender roles while at the same time clinging desperately to them.