Drilling into noise

The science of statistics is all about differentiating signal from noise. This exercise is far from trivial: Although there is enough computing power in today's laptops to churn out very sophisticated analyses, it is easily overlooked that data analysis is also a cognitive activity.

Numerical skills alone are often insufficient to understand a data set—indeed, number-crunching ability that's unaccompanied by informed judgment can often do more harm than good.

This fact frequently becomes apparent in the climate arena, where the ability to use pivot tables in Excel or to do a simple linear regressions is often over-interpreted as deep statistical competence.

The graph below illustrates this problem with the global temperature data: although there is no question that the trend is increasing, it is always possible to cherry pick periods for analysis during which there is no significant increase in temperature. Of course, those “analyses” are a meaningless distraction from what is actually happening on our planet (the only one we've got, by the way).

Similar comments apply to some of the analyses reported in the blogosphere of our recent data on rejection of science and conspiracist ideation. We have already dealt with the "scamming" issue here and here, and we will not take it up again in this post.

Instead, we focus on the in-principle problems exhibited by some of the blog-analyses of our data. Two related problems and misconceptions appear to be pervasive: first, blog analysts have failed to differentiate between signal and noise, and second, no one who has toyed with our data has thus far exhibited any knowledge of the crucial notion of a latent construct or latent variable.

Let's consider the signal vs. noise issue first. We use the item in our title, viz. that NASA faked the moon landing, for illustration. Several commentators have argued that the title was misleading because if one only considers level X of climate "skepticism" and level Y of moon endorsement, then there were none or only very few data points in that cell in the Excel spreadsheet.

Perhaps.

But that is drilling into the noise and ignoring the signal.

The signal turns out to be there and it is quite unambiguous: computing a Pearson correlation across all data points between the moon-landing item and HIV denial reveals a correlation of -.25. Likewise, for lung cancer, the correlation is -.23. Both are highly significant at p < .0000...0001 (the exact value is 10 -16, which is another way of saying that the probability of those correlations arising by chance is infinitesimally small).

What about climate? The correlation between the Moon item and the "CauseCO2" item is smaller, around -.12, but also highly significant, p < .0001.

Now you know why the title of our paper was “NASA faked the moon landing—Therefore (Climate) Science is a Hoax: An Anatomy of the Motivated Rejection of Science.” We put the "(climate)" in parentheses before "science" because the association between conspiracist ideation and rejection of science was greater for the other sciences than for climate science.

(As an intriguing aside, by the logic that's been applied to our data by some critics, the larger correlations involving other sciences would suggest that AIDS researchers—keen to get their grants renewed?— scammed our survey to make AIDS deniers look bad.)

But we can do better than extract a signal by simple correlations.

Far better.

This brings us to our second issue, the role of latent variables.

To understand this concept, one must first consider the fact that on any cognitive test or survey, any one item, however well designed, will not provide an error-free measure of the psychological construct of interest. No single puzzle can tell you about a person’s IQ, no single question will reveal one’s personality, and no single moon landing will reveal a person’s propensity for conspiracist ideation.

So the correlations we just reported constitute a better signal than the noise that overwhelms a selected few cells of an Excel spreadsheet, but they are still "contaminated" by measurement error or item variance—that is, the data reflect the idisosyncracy of the particular item in addition to information about the construct of interest, in this case conspiracist ideation.

SEM is a technique that estimates latent constructs—that is, the hypothesized psychological construct of interest, such as intelligence or personality or conspiracist ideation. SEM does this by considering multiple items, thereby removing the measurement error that besets individual test items.

We cannot get into the details here, but basically SEM permits computation of the error-free associations between constructs, such as one's attitudes towards science and one’s conspiracist ideation. It is because measurement error has been reduced or eliminated, that correlations between constructs are higher in magnitude than might be suggested by the pairwise correlations between items.

And because SEM removes measurement error, the associations between constructs are particularly resilient, as we showed earlier when all observations are removed that might conceivably represent “scamming.”

When the long-term temperature trend is ignored in favour of a few years of declining temperatures after a unique scorcher, this is missing the statistical forest not just for a tree but for a little twig on a tree.

Likewise, when the associations between latent constructs in our data are ignored in favour of one or two cells in an Excel spreadsheet, that’s missing the statistical forest not just for a little twig on a single tree but for a single leg of a pinebark beetle on that twig that’s eating its way northbound through the Rockies as the globe is warming.

Nature abhors a straight line, forty years tells us only one thing - that we've had inconsequential warming over that period. Hardly surprising in a post glacial pre glacial period and it certainly oes not conclude that the warming will continue, stop or reverse. So why the panic?

The signal turns out to be there and it is quite unambiguous: computing a Pearson correlation across all data points between the moon-landing item and HIV denial reveals a correlation of -.25.

This sounds much more meaningful until you realize you get a similar negative correlation even if nobody believes the moon landing was faked. That's right. Remove the ten responses that agree or strongly agree the moon landing was faked, and you get a correlation of -.21. And if you only remove the two most conspiratorial responses (the ones many people say are fake), the correlation is -.23.

The same pattern happens with "the Moon item and the 'CauseCO2' item." The full data set gives a correlation of -.12. That drops to -.08 if you remove the two most conspiratorial answers. It then only drops to -.07 if you remove every answer claiming to believe the moon landing was faked.

Keep this in mind when hearing about correlations in this post. You can find correlations between people's belief in a conspiracy and other things even if nobody believes in the conspiracy.

This is an assumption, but I believe the the reason you find the title funny is because it plays on your preconceptions about people who reject climate science as opposed to your preconceptions about people who believe the moon landing is hoax. I doubt if you would laugh to your self and say "oh ho ho, those nutty moonlanders, they probably don't even believe climate science!" But the authors know you *do* laugh to yourself and say "ho ho those nutty climate deniers probably don't even believe in the moon landing". The title implies, and the "humor" relies upon, a generalization about climate skeptics rather than moon landing skeptics.

So there you go. You misinterpretted what I meant by the implication. The implication of the title is backwards from what the paper shows, and if you interpret it literally (as in forward), the statement is still false.

This is tied up with the apparent confusion regarding exactly what group the paper is about. Is it about science blog visitors, climate deniers, or conspiracy theorists? In the text of the paper it's stated that the survey had N=1163 (or some number like that). That would only be true if the paper were about the attitudes of science blog visitors. You could say something about the attitudes of science blog visitors with the kind of certainty implied by an N=1163 survey, like how many of them deny climate science and how many of them embrace conspiracy theorists with some strength. But if the paper is actually about climate deniers, then the N is much lower, maybe 150 or even lower depending on where you draw the line between denial and skepticism. If it's about people who are prone to easily accept conspiracy theories then the N is much lower still. There might be 10-20 people in the survey who show a propensity to embrace large numbers of divergent conspiracy theories. The sample number is too small to make and strong statements about how conspiracy prone people think or what other types of ideas they're likely to embrace. So if the title is meant to say something about how conspiracy people think rather than how climate deniers think, the survey is a victim of flawed methodology. Lewandowsky et al should have survey conspiracy theorists rather than the climate interested. Unfortunately they botched.

Or, tell the truth that this survey was meant to say something about climate deniers in general *not* conspiracy theorists, and what it wanted to say is that they are prone to believe heavily in conspiracies. Unfortunately for the authors, that's not what the data shows.

I think the text of the original article is fairly factual and follows from the data (to the extent that we can assume the data is usable). The title on the other hand, says something that's not supported.

@Brandon (currently #70) - I'm not sure how you got exactly the numbers you got, but I believe that's exactly the point. Lewandowsky and co-authors weren't binning groups into conspiracists and non-conspiracists or "warmers" vs "deniers", they were measuring people's mental models of the world, and each of us falls somewhere on a spectrum. Those who are willing to marginally entertain the moon-landing conspiracy theory (by not indicating "strongly disbelieve") are a much larger group in the sample than just those who say they "believe". And that measures something.

It would do well if - in this case as in many others - people who wanted to criticize a piece of scientific work put in their own effort first to actually understand what was done and what it means. It sounds like some folks are very slowly getting there... well, it should be an education in a little bit of psychological science on the way I suppose, for those will to wade through and ignore the many loud voices of confused ignorance that have been on display here...

Well, one thing's for sure, Carrie: if nothing else the title is provocative and has certainly focused attention on the topic of the paper. I think we've seen that demonstrated beyond doubt, so the title is a success. Thanks to this brouhaha much more to come from the authors' colleagues in the field, I'm sure.

Really, thanks to the hue and cry from outraged laypersons Lewandowsky's paper should serve as a kind of heads-up. It's not the first to put "climate skeptics" under a microscope, after all.

There's going to be a lot of the kind of back-and-forth over angst caused by investigations of the cognition failures leading to paralysis over climate change. Unless researchers can somehow be persuaded not to follow their curiosity, we can look forward to learning much on the topic.

It was inevitable that scientific attention would be aroused by such a curious phenomenon as the so-called "climate debate." We've seen some littler versions of the same general picture in the past but never with the broad scope of the present battlefield; tetraethyl lead was the subject of a multi-decade struggle but the fracas was confined mostly to insiders.

A fascinating thing; geophysics moves on, littering its path with unexpected and intriguing features of human nature.

As a follow-up to my earlier comment, I decided to have some fun and see what correlations exist for people who believe in the moon landing. To do this, I extracted the ten responses that claim to agree the moon landing was faked. I then tested the correlation between the same two item pairs as this blog post mentions.

CYMoon - CauseHIV: .64.
CYMoon - CauseC02: .46

Now then, because of the small sample size, the significance levels are much lower than in the blog post(95% and 80%). Moreover, this isn't an actual test of anything. It's just a demonstration of how the demographics of data allows one to make claims that seem reasonable but aren't actually supported by the data.

As my two comments show, the primary reason for the negative correlations discussed in this post has nothing to do with people who actually believe in the conspiracy being discussed. Put bluntly, people who don't believe in a conspiracy are being used to paint other people who don't believe in a conspiracy as believing in a conspiracy.

apsmith you say you are "not sure how you got exactly the numbers [I] got." I find that interesting as I did exactly what was done in this blog post, just having removed the ten responses I said I removed. I even tested to make sure I was doing the same thing by replicating the post's results.

As for it being "exactly the point," I find that hard to believe. You say the paper discusses people "who are willing to marginally entertain" conspiracy theories, but I can't find a single thing in the paper that says that. Instead, I find a great deal of discussion talking about belief in conspiracy theories. In other words, I find a great deal of discussion of what I refer to and none of what you refer to.

This makes it peculiar for you to say "people should put in their own effort first to actually understand what was done and what it means."

Brad Keyes @25, you may wish to distinguish between the claim that "There was a concerted effort to undermine the science of climate change", which was true, although it is no longer; and the claim that "all vocal opponents of global warming science are funded by the fossil fuel industry" (which is false). I note that some members of the fossil fuel industry, notably the Koch brothers continue to be active funders of global warming pseudo-science.

I need to take the values in my second comment. They seemed unusually high to me, so I took a second look at them. It turns out when I copied the ten rows, I didn't copy them into an empty variable. I guess I wasn't paying enough attention because originally wasn't interested in doing anything with those responses. When I made sure to copy the data correctly, the results had no significance.

The results in my first comment are unaffected, and my point remains the same. Still, I'm sorry about posting results without checking them.

Carrie, you're spot on about finding the title amusing "because it plays on your preconceptions about people who reject climate science " in my case, certainly. It doesn't make me think any less of people I know well who can't accept the science, because I know all their other attributes as well.

However, when I first come across a person, if one of the first things I learn about them is that they reject an entire body of science, well I must admit it colours my perception in other ways. That's mental models for you. (Most people use stereotyping to some degree, it's easier on the brain!)

(Not saying the first thing that comes to mind is "I wonder if they think the moon landing is a hoax". But it wouldn't surprise quite as much if I found out they did.)

To the moderator: Your snipping of Carrie N's title at 34 pretty much proves their point. No doubt the first part of the title of Lewandowsky 2012 (in press), if found in a comment would also, and rightly be snipped as inflammatory.

Sou @42, the direction of causation conspiracy theorist -> AGW "skeptic" correctly represents the findings of the paper. The use of "therefore" in the title, however, indicates that that is supposed to be a logical inference. That is not supported by the paper, and is not reflective of the reasoning of any person I am aware of, or (I believe) any real person.

It is very difficult to believe that the title is anything other than a deliberate attempt to be offensive so as to draw attention to a paper of poor quality, but which is thought to be useful for "messaging" in the climate wars. Steve McIntyre has incorrectly attempted to infer a moral condemnation of Lewandowsky from certain of my comments (now corrected). Let me leave no-one in any doubt. In choosing the title of his paper, Lewandowsky not only acted unscientifically, but immorally as well. It was a despicable act.

apsmith @71, that that shows is that the results could just as easily be interpreted as saying that "skeptics" are less willing to say that an empirical claim is "absolutely false" rather than "probably false". In that I am bang along side them. When I realized the actual responses available, I realized that for all conspiracy theory items, my response would have been "probably false" (2) even though I have no inclination to accept any of the conspiracy theories presented (ie, I consider them all false except for one for which I do not have relevant evidence).

Lewandowsky would interpret that as meaning I am inclined to accept conspiracy theories. Personally I think it only means that I know what the word "absolute" means.

Carrie, it's a joke, intended to convey the paper's finding that -some- people have a tendency to substitute fables for facts and that this fallibility can be found spanning different topics.

So now the truth comes out. Or at least the next version of the truth.

Its all just a big joke son, nothing to see here. We're just funnin' you boy. It's just 'ha ha' funny .. when a connection is discovered no matter how slight, the humor potential becomes irresistible -we just can't risk making fun of people

We keep getting told this is a supposedly legit scholarly paper on a serious subject.

Which is it - legitimate research or a big joke?

I wonder if the grant folks who paid big money for this for this big fun think its so funny?

It isn't all sciencey and stuff, and I probably got some stuff wrong, being a mere blog denizen and such, but was written in the same "calm down, its its a joke son" mindset as exhibited by the new spin shown by supporters in this thread.

If the authors think it acceptable to use a humorus joke as the title of a supposedly serious work, then its only fair we all get to join in the fun.

@ Tom Curtis, I respect your right to be outraged and for you to have your own interpretation. Different people respond differently to humour. Everyone has their own mental models. (I reserve my outrage for other matters :D)

I'm more in tune with Doug Bostrom on this one as far as I understand his posts on the subject (and, presumably, Prof Lewandowsky and his colleagues.)

I know there are a lot of "skeptics" commenting here disputing both the veracity of the article and the way comments are handled here, so I would like to point out that I currently have two comments in threads over at Climate Audit that have been held in moderation for days. One of these is a comment on censorship at WUWT that is being held in moderation in the thread on censorship. The other is a comment simply disputing technical points of Mcintyre's analysis of Lewandowsky's supposed "fake" results (an analysis which is wrong and misleading).

I think a lot of "skeptics" here probably believe that their "side" of this "debate" argues in good faith, but as someone who has seen completely non-inflammatory, innocuous comments about scientific points deleted from WUWT and disappeared down the moderation hole at Climate Audit, I would suggest that those accusing this blog's owner of egregious moderation should look to the mote in their own eye.

No, you may be right that McI himself doesn't explicitly reject the overall science. Those on his threads sure do tough

That claim is without merit. I'm one of those you paint with your broad brush. And I don't reject the overall science.

And I - along with a large number of others far more talented and knowledgeable than I am - are all making a concerted, legitimate effort to review the results claimed in this paper.

When the "believers" - the supporters of this work - issued the all to typical 'if you don't like it - the questions are all in the paper - go do it yourself' challenge - I invested a large amount of time and effort and did exactly that.

And in literally a few days was able to accomplish what Lewandowsky claims he could not - obtained a response from the skeptics, both high in quality and in numbers that dwarf the original response.

The signal turns out to be there and it is quite unambiguous: computing a Pearson correlation across all data points between the moon-landing item and HIV denial reveals a correlation of -.25.

This sounds much more meaningful until you realize you get a similar negative correlation even if nobody believes the moon landing was faked. That's right. Remove the ten responses that agree or strongly agree the moon landing was faked, and you get a correlation of -.21.

Brad ... thank you - exactly what I've been saying all along.

But oi course the question is largely moot now it would seem as the latest position seems to agree, that even though there was little, or as you show - no, significant association found, use of the sensationalized title was simply a good old joke.

...was able to accomplish what Lewandowsky claims he could not - obtained a response from the skeptics, both high in quality and in numbers that dwarf the original response.

Not skewed at all, either! The skeptic's survey was launched and run in an entirely calm atmosphere, enrolling respondents who were entirely unaware of the reason for the hasty assemblage and launch of the "research."

I'm still wondering why the provenance of skeptic responses is so important. Is the "climate skeptic philosophy" something with an exponential decay function against distance from a "climate skeptic blog," akin to gravitation? Surely the thinking is robust to displacement?

Out of curiosity, with regard to the title of the post - are people more outraged on behalf of the moon hoax fraternity, who seem to be silent on the matter here and have no advocates in this thread. Or are they more outraged on behalf of those who are unconcerned (or worse) about arguably the greatest issue affecting humanity this century?

Well, one thing's for sure, Carrie: if nothing else the title is provocative and has certainly focused attention on the topic of the paper. I think we've seen that demonstrated beyond doubt, so the title is a success. Thanks to this brouhaha much more to come from the authors' colleagues in the field, I'm sure.

Real science uses real science to focus attention on a paper and be "provocative" ... not funny jokes.

It isn't science if the results alone are insufficient to gain critical attention.

If you are unfamiliar with the theory and methods used perhaps studying latent variables and structural equation modelling so that you can write your own code would provide a more meaningful replication of the study, or refute it, than claiming that Prof L has an obligation to provide the computer code or scripts used which looks a little like wanting to 'borrow' someones homework.

The link in the essay above gives some useful links for further study.

@ izen - dare we suggest he even go to the extent of designing and conducting his own independent research? You know, develop a research project, work out what he wants to test, read the relevant literature, design a survey instrument or whatever means he wants to use to test it, do the research and analyse it.

Just to let you know that a couple of those links seem to have got messed up - the second one leads to a google warning. (Could be restricted entry). The third link goes to what looks like a home page of Prof Greenlee.

And I - along with a large number of others far more talented and knowledgeable than I am - are all making a concerted, legitimate effort to review the results claimed in this paper.

While you do seem to be making such an efforted, Mcintyre is censoring anyone who disagrees with him through jiggery-pokery with the moderation queue, refusing to engage with my criticism of his analysis of "fake results" and failing to provide the basic statistical tests necessary to confirm his hypothesis, because he knows they'll show the opposite of what he claims. So no, I don't think you can make the claim you've made about "legitimate effort."

patg, I can't post the moderated comments directly because I can't view them from this IP address (I posted them from a different location). There's an outline of what happened and my previous experience with these "legitimate" inquirers at my blog.

Moderator Response:

In the spirit of openness and transparency, an explanatory note: Mr. McIntyre on the linked thread at the above link is exceedingly wrong on multiple accounts:

1. Stephan plays no role in moderating the threads dealing with his paper.

2. Stephan has no say nor input in the moderation of these same threads.

3. Mr. Fuller was repeatedly counseled to cease posting inflammatory comments (and complaining about moderation) to better bring his comments into compliance with the publicly-posted Comments Policy (an occurrence Mr. McIntyre is personally familiar with), including being given a final warning. More than one inflammatory comment was then subsequently posted by Mr. Fuller, forcing the revocation of his commenting privileges by the moderation staff, an action they find distasteful to do but unavoidable in this instance. However, an unfortunate byproduct of the moderation software results in all of the comments by said individual also being deleted along with their commenting privileges being suspended. Had Mr. Fuller brought his comment construction into compliance with the Comments Policy, a condition of posting few other individuals find onerous, then his presence would be still welcome here.

Discussion of moderation is still off-topic; please return to the topic of this thread, and thank you for your forbearance in this manner.

There is a long tradition of humorous (whether successful or not) titles for papers of this sort. I remember doing a lit review of papers on female attractiveness to males in an intro Sociology class 30 years ago in college. The title of one of the papers I reviewed? "The Girls Really Do Get Prettier at Closing Time". Take a guess at what was studied.

#107 faustusnotes, when I post comments, i first write a Word document and save it. I do that for several reasons: The upload fails, It goes into endless moderation, Or the moderator over edits. Do you have a copy of your comments that went into endless moderation?

I don't know why moderation at other blogs would be being discussed here, but since faustusnotes made an issue of it, I feel I should point out in the past few days, I've had several comments wind up in moderation for extended periods at Climate Audit. In fact, I have one that has been in moderation for half a day now. None of them are things Steve McIntyre disagrees with or has reason to censor. And yes, he did fish other comments out/make at least one new post while leaving comments of mine in moderation.

I think it's hard to argue comments that get flagged as spam are being intentionally held in moderation. That's possible, but it's also possible they've just been overlooked. Moderating can be very tedious.

Moderator Response: With that said, moderation on any website is now off-topic here.

Those who think there have been organisations funded for the purpose of spreading disinformation about climate change (in whole or part) might or might not be doing so on little evidence.

Lets say they are just for argument's sake. In that case, proponents might adopt their position because they prefer that explanation to any other that could explain why US citizens are less well informed on the matter than those of any other western nation. (eg How could a nation as large and prosperous as the USA have neglected its education system so badly, and it would be implausible that US citizens are intrinsically dumber than everyone else in the world).

On the other hand, it could be that there is good evidence to support the theses that certain individuals and organisations not only donate funds to promote particular ideas, but that organisations posing as 'grass roots' organisations have been set up for the purpose. Eli Rabett said he reads IRS Form 990s for chuckles - you could ask him about this. (AFAIK the USA is unique in its SuperPACS and in declaring that corporations are people.)

Brad, I suggest you read this post. It explains why your interpretation is wrong.

But to answer your question, the paper showed that there was a statistically significant increase in the male perception of female attractiveness as patrons (male) hung around (read: got more drunk). Just as this paper showed that there was a statistically significant negative correlation between believing in conspiracy theories and acceptance of climate science. The correlation is not strong (it is not predictive) in this study, but it is there and it is significant.

Just as this paper showed that there was a statistically significant negative correlation between believing in conspiracy theories and acceptance of climate science. The correlation is not strong (it is not predictive) in this study, but it is there and it is significant.

If the correlation is not strong how is it significant?

And how is it a correlation at all, if as Brandon noted earlier the correlation exists whether the item is included or is not?