Our lives are full of findings from research results. Not all of the claims they make are valid. Some studies use samples that are not representative. Other studies make claims when they have no comparison group. The question wording or even question order can bias survey results. How do you know when you can trust the research claims? What do you need to look for?

This talk will give you tips to judge the research you read. We look at examples from the current news and discuss what you need to know to judge whether the research finding can be trusted. We will discuss the threats to validity for each study and whether the research meets the standard of reproducibility. That is, could another researcher possibly reproduce this finding?

Fran Featherston is retired from the U.S. federal government where she worked 27 years, as a researcher at the National Science Foundation (10 years) and the Governmental Accountability Office (17 years). Her areas of expertise are survey research, research design, and designing methods that are user friendly. She also worked for the State of Washington's court systems so that she has experienced the challenges of designing high quality research for all three branches of the government.

2014 NCAS Philip J. Klass Award PresentationOn November 15, NCAS president Walter Rowe presented the 2014 NCAS Philip J. Klass Award for outstanding contributions in promoting critical thinking and scientific understanding to Steven Salzberg, Professor of Biomedical Engineering, Computer Science, and Biostatistics in the Institute of Genetic Medicine at The Johns Hopkins University School of Medicine. In accepting the award, Dr. Salzberg said, "It's important that as skeptics, all of us try to communicate, not only with people who already agree with us, but with people who don't agree with us. I've tried to learn skills to convince people to think skeptically. I say, 'Why do you think that [treatment] is going to help? Where did you hear that? Have you had any experience with this? Has this worked for you?' and try to get them to think skeptically."

Torn From Today's HeadlinesBy Scott SnellThe American University Hosts a UFO Panel of 4 Believers and 0 SkepticsThe American University hosted a panel discussion, "UFOs: Encounters by Generals, Pilots, and Government Officials" on the evening of November 12, 2014. Presented in conjunction with the AU Honors Colloquium in Social Sciences course, "Alien Contact: Science and Science Fiction," the students, faculty, and public in attendance heard notably one-sided presentations from four proponents of UFOs as "real, physical phenomena appearing to be under intelligent control."

Moderator Patrick Jackson, Associate Dean for Undergraduate Education in AU's School of International Service (filling in for PBS science correspondent Miles O'Brien, who had to cancel "because of another assignment"), said in his opening remarks that a "university is at its best" when encouraging "civil discussion" and "healthy skepticism."

Yet what followed was sadly lacking in important information and skeptical perspectives regarding UFOs. Panelists Leslie Kean (journalist and author of the 2010 bestselling book, UFOs: Generals, Pilots and Government Officials Go On the Record) and Richard Haines, PhD (experimental psychologist and former visual perception researcher at NASA-Ames Research Center) are researchers with pronounced "blind spots" towards evidence that would contradict their viewpoint that some UFOs are probably alien craft.

Panelist Colonel Charles Halt (USAF, retired), claims to have seen extraordinary UFOs in 1980 near Royal Air Force Station Bentwaters (UK), where he was deputy base commander. (This is typically known as the "Rendlesham Forest Incident.") Rounding out the panel was Thomas Carey, coauthor of the 2007 book, Witness to Roswell: Unmasking the 60-Year Cover-Up, which asserts that an extraterrestrial craft crashed near Roswell, NM in 1947.

Even savvy audience members (if unfamiliar with the full story about UFOs) were probably impressed by Kean's and Haines' presentations. As UFO skeptic Robert Sheaffer noted in his review of Kean's book, "It's easy to tout UFO cases as having no conventional explanation as long as you completely ignore everything that's been written to the contrary."

For example, I was surprised to see Kean include a 1971 "physical traces" case from Delphos, Kansas, in which an allegedly witnessed UFO left a whitish ring on the ground, apparently necrotic and showing little if any water absorption as compared to soil inside and outside the ring. Analysis of the soil (as reported a few years later by Jacques Vallee, a prominent non-skeptical UFOlogist) indicated the presence of a fungus-like organism, capable of producing a phenomenon called a "fairy ring."

When I pointed this out to Kean during the audience Q&A, she replied (in a rather halting, uncertain voice) that she had only recently included this case in her presentation, hadn't looked closely into it, said she didn't know about the fairy-ring fungus finding, and closed by thanking me for telling her.

The highlight of Kean's presentation was a photo taken in 1971 by a mapping aircraft of the Costa Rican government. Though an intriguing photo, an optical-effects explanation has not been ruled out (see the Rocky Mountain Paranormal Research Society's article at http://www.rockymountainparanormal.com/1971costaricaufo/index.html). (For future research, I suggest that optical experts from the camera's manufacturer, Carl Zeiss AG, be consulted. They may have records of any similar effects reported by users, or could recognize the effect if seen during optical tests they performed on their product.)

Both Kean and Haines said that solving the UFO mystery requires U.S. government participation in an international scientific research effort. Neither mentioned the 1960s U.S. Air Force-funded University of Colorado scientific study of UFOs, which resulted in the "Condon Report." (The report is hosted by NCAS at http://files.ncas.org/condon/ with permission of the Regents of the University of Colorado.) The report concluded that "further extensive study of UFOs probably cannot be justified in the expectation that science will be advanced thereby."

Haines' presentation, "Unidentified Aerial Phenomena [his preferred term for UFOs] and Flight Safety," focused on incidents involving claimed flight control effects and/or instrument effects during pilot UFO sightings. The cases, as presented by Haines, appeared to be remarkable and worthy of further study (though of course there may be more to those stories than was presented). Haines plausibly suggested that commercial airline pilots who have seen a UFO are likely to not report it, suggesting that we're only getting "the tip of the iceberg" for this type of reports. This presentation probably connected well with the audience, who likely have a lot of confidence in the trustworthiness, responsibility and qualifications of pilots, and also probably considered Haines an esteemed authority based on his NASA credentials.

Regarding pilots as UFO eyewitnesses, both Kean and Haines disregard the conclusions of a UFO authority they otherwise cite frequently, Dr. J. Allen Hynek. In The Hynek UFO Report (Dell, 1977), he wrote, "Surprisingly, commercial and military pilots appear to make relatively poor witnesses." (See James Oberg's article, "Case Studies In Pilot Misperceptions Of 'UFOs'", at http://www.zipworld.com.au/~psmith/pilot-ufos.html.)

The remaining two presentations were probably not of much interest to discerning audience members: Colonel Halt told the latest version of his UFO sighting, and Thomas Carey tediously rehashed the Roswell UFO myth. Carey may have sensed that his presentation needed a boost, because he promised never-before-heard news to come at the end of his talk.

WTOP (apparently the only news organization that covered the event) of course led its story with Carey's Roswell "smoking gun":
"'We have come into possession of a couple of Kodachrome color slides of an alien being lying in a glass case,' author and researcher Thomas Carey told the near-capacity crowd [actually about 3/4 full] in Abramson Recital Hall.

'We took it to the official historian of Kodak up in Rochester, New York, and he did his due diligence on it, and he said yes, this filmstrip, the slides are from 1947.'

Carey says he plans to reveal the images early next year."

WTOP was similarly impressed by Halt's anecdote, calling it "the most riveting presentation of the night," unaware that Halt's story has gotten more far-fetched over the years. (See Ian Ridpath's analysis at http://www.ianridpath.com/ufo/Halt_affidavit.htm. Ironically, during the Q&A, Halt complained that it's the stories of fellow witnesses, who dispute his fanciful version, that keep changing.)

Carey's "smoking gun" did not go over so well with at least one of the panelists. Kean later wrote on her Facebook page, "...you are not the only one who has been impacted by the Roswell part of the AU conference dominating the media coverage. [...] It was made worse by Carey's surprise announcement. I hope that the people on this page can see through that and realize how positive it was that the panel was hosted by a major university, and that's what's important."

Carey also appeared to unwittingly impugn the reputations of UFO "abductees." During the Q&A, Carey fielded an audience question about George Adamski (a "contactee" who became famous in the 1950s for claiming to ride with UFO aliens to other planets), and with his carelessly phrased answer, probably unintentionally lumped contactees and abductees disreputably together in the minds of any UFO novices in the audience with his disparaging dismissal of contactee claims as setting back "serious" UFOlogy.

After the program, I spoke with a few of the honor students who are taking the "Alien Contact" course. They've been reading Kean's book (the centerpiece of the course) and consider it an excellent resource on UFOs, as would be expected from the course description: "This course is about mankind's desire for extraterrestrial contact. It examines the historical links between film and television to historical world events as they relate to increased reports of contact. Students read and study about the Roswell Incident in 1947, the Phoenix lights in 1997, and other UFO phenomena revealed by the recent declassification of international government files on UFO sightings. The class sees classic science fiction films from the 1950's to the present and reads the novels that spawned them as well as examining television shows that have become part of our 'alien pop culture' for fifty years..."

I also spoke with course instructor John Weiskopf, who typically teaches film/screenwriter courses at AU. Given that his course covers the intersection of UFO reports and science fiction, I asked him if he'd included the research of Martin Kottmeyer ("The Eyes that Spoke," probably the clearest example of science fiction shaping UFO folklore) for his students. Weiskopf told me he knew of Kottmeyer's work, but apparently had forgotten about it, and enthusiastically agreed he should go over that with his students!

When I complained to Weiskopf about the lack of skeptics on the UFO panel, he summoned Michael Manson, director of the AU Honors Program. Manson asked me who should've been invited, as if they had no idea where to begin. Is it too hard to try Googling "UFO skeptic" for a start?

In October 2013, Amazon launched the AmazonSmile Foundation, which allows customers to support their favorite charitable organizations when shopping at Amazon.com, at no added cost. The AmazonSmile Foundation will donate 0.5% of the purchase price from eligible AmazonSmile purchases to the customer's designated 501(c)(3) public charitable organization. NCAS has registered as a participating organization.

Shadow Light
Some members and contacts of NCAS receive a postal notification of this and every new monthly Shadow of a Doubt. The Shadow Lightpostcard announces the monthly lecture and highlights of the electronic Shadow of a Doubt, which is available online at ncas.org/shadow. NCAS thereby reduces Shadow production and postage costs. To further reduce costs, members and contacts can opt out of postal notification altogether, while continuing to receive Shadow of a Doubt via e-mail. To opt out, send us an e-mail at ncas@ncas.org.

Drinking Skeptically in MD and VA!

On Wednesday, December 10 at 7:00 p.m., please join fellow NCASers at either of our simultaneous DC-area Drinking Skeptically events:

The most recent Washingtonian magazine list of the best DC-area bars (2013) includes Jackie's Sidebar: "Insider tip: Sop up the spirits with some of the best bar snacks around. We love the Chicago-style dog and grilled-seaweed-sprinkled popcorn."

Drinking Skeptically is an informal social event designed to promote fellowship and networking among skeptics, critical-thinkers, and like-minded individuals. There's no cover charge and all are welcome. Don't drink? Don't let that stop you from joining us! Some of the world's most famous skeptics are teetotalers, and we are happy to have you! Remember that drinking skeptically means drinking responsibly. If there's one thing science has taught us, it's the effects of alcohol on the human body.

Skeptic Line Number
As of March 2013, NCAS has switched telephone numbers, from 301-587-3827 to 240-670-NCAS (6227).

Time to Renew?
Be sure to check your renewal date above your postal address on the Shadow Light postcard. Send any queries to ncas@ncas.org. Use the online membership form to renew.