How Many Times Must We Bury Behaviorism?

I noticed continued resistance to survey evidence in the comments. Frankly, it's hard to believe there's anyone left in social science who doesn't take surveys seriously. Friedman's original arguments against survey evidence were half-baked, and no one's come up with any better objections since.

Yes, there are problems with surveys, but there are problems with all empirical evidence. And most of the data that economists use "instead of" surveys are themselves collected using... surveys! Where do you think unemployment numbers come from - hiring private eyes to follow people around all day? For goodness sakes, even if we did hire P.I.s, the data would reflect what the P.I.s said people did. It would be a second-hand survey, but a survey nonetheless.

Comments and Sharing

It depends what the survey data is for. If you are studying opinion per se--e.g. Presidential polling--then it's pretty good. Your studying opinions by simply asking for opinions.

For income and other basic econ data, I think it's overrated. Comparing IRS data to BLS data on income, for instance, shows big differences. Most people are too stupid and/or ignorant to give accurate answers.

I'd say it's the most unreliable about questions of motivation: asking WHY people do what they do.
People don't know themselves that well.

No-doc home loans and home loans where someone produces income documentation are both sorts of surveys. Bryan might say the latter is simply surveying what a company paycheck says about someone's income, so it's only a second-hand survey. And yet one is considerably more accurate than the other.

For goodness sakes, even if we did hire P.I.s, the data would reflect what the P.I.s said people did.

Yes, but the motivation for lying is somewhat different in the two cases.

Simply because both are surveys and because it's impossible to avoid all bias doesn't mean that some are worse than others.

Friedman’s half-baked and Bryan overcooks. For a good, simple overview of social research with pros and cons of not just doing surveys but different types of surveys see: Alan Aldridge & Ken Levine’s Surveying the Social World: Principles and Practice in Survey Research.

I think the design of the survey is key. Whenever presenting survey evidence, I think the burden of proof always falls on the one citing the survey to show that it is incentive compatible. We could, of course, start from the assumption that people tell the truth unless they have a particular incentive to lie; but a more realistic assumption would probably be that people lie unless they have a particular incentive to tell the truth.

"A more realistic assumption would probably be that people lie unless they have a particular incentive to tell the truth."

Too cynical. Most people have at least some moral qualms about lying and wouldn't do it for no reason. People don't have to answer surveys, anyway. If they do it, it's typically because they like talking about themselves and/or they want to serve the cause of science. Either way it's a reason to tell the truth.

A bigger problem is that questions are often poorly framed and data are often misinterpreted.

"but a more realistic assumption would probably be that people lie unless they have a particular incentive to tell the truth."

Think through what you're saying: for surveys that ask who a person is going to vote for -- McCain, Obama, or Undecided -- are you claiming that unless the survey is meticulously designed, those who say McCain really will vote for Obama, those who say Obama really will vote for McCain, and Undecideds are certain who they'll vote for?

Indeed for a two-answer question, which many survey questions belong to, the "lie as default" strategy wouldn't work -- the recorder would just code your response as the other possible answer.

You must not talk much to people if you think that "lie as the default" is what most people in most situations have.

I read Bryan's post explaining why the ideas were "half-baked" as admitting nearly every criticism I've heard of surveys. Apparently Bryan also believes that "it's hard to believe there's anyone left in social science who does take surveys too seriously," as well, considering how he views it as extremely obvious how and when to take survey results with a grain of salt.

And yet, far too often I read stories or papers where social scientists uncritically report survey data when there are enormous problems with self-interest in the results. Since social scientists so often trust survey data even in the cases where Bryan concedes that it's flawed, skepticism is still entirely warranted.

Even with the general skepticism toward surveys, I'm baffled that anyone is questioning their use in this particular example. Bryan uses survey data to show that people support social security, including young people. And every effort to reform social security (including in the favor of young people) has been resoundingly defeated, even when conservatives ran everything. With all the complaints, you'd think the data doesn't completely and totally conform to our experiences of how people actually vote or something.

it's hard to believe there's anyone left in social science who doesn't take surveys seriously

And it's hard to believe there's anyone outside social science who takes the majority of them seriously, beyond anything more sophisticated than "who are you going to vote for?".

I've seen way too many surveys that reek of bias, in order to support the "progressive" liberal left agenda. And they get reported by the MSM like they're The Clay Tablets.

Europeans, for example, are "happier" than Americans. Remember that one, from not so long ago? The only "Americans" I know who believe that are liberals (the same people who tell me that, in spite of clearly different economic growth rates, people in Europe are just as rich and have just as many opportunities in life as Americans, on average -- I don't believe it). The outcomes of so many surveys depend almost entirely, on both what is asked and how it's framed.

Very, very few surveys are designed by libertarians. Which is a key reason this libertarian tends to doubt the validity of many surveys.

Having worked in Market Research, I've seen how powerful surveys can be if carried out correctly, but also how misleading they can be if not (usually due to faulty survey design). Many times I find myself asking what questions (or answers) are left out just as much as what was included.

I find it hard to doubt the accuracy of the social security survey very much. I didn't see the questions themselves, but even if it were off by several percentage points it wouldn't change the outcome, and the arguments against its use were all abstract.

Lately I've become more and more enamored with prediction markets. Ask people to put their money where their mouth is, and you start seeing much more honesty.

Blogging software: Powered by Movable Type 4.2.1.
Pictures courtesy of the authors.
All opinions expressed on EconLog reflect those of the author or individual commenters, and do
not necessarily represent the views or positions of the Library of
Economics and Liberty (Econlib) website or its owner, Liberty Fund,
Inc.

The cuneiform inscription in the Liberty Fund logo is the
earliest-known written appearance of the word
"freedom" (amagi), or "liberty." It
is taken from a clay document written about 2300 B.C. in the Sumerian city-state of Lagash.