Articles and Analysis

My post on methodological disclosure by pollsters earlier this week reproduced list of "minimal (Level 1)" information that members of the National Council on Public Polls (NCPP) are required to include in all public reports, including "fieldwork provider (if applicable)." That means that if a pollster subcontracts telephone interviewing to a call center, they are supposed to disclose the name of the company in their reports.

Yesterday, I received an email from a media pollster who asks a reasonable question:

I've never understood why a firm would be required to disclose the fieldwork provider. I cannot see how it reveals anything about the quality of the poll, unless it is done by some firm everyone knows to be unreliable. To me, it is sort of a trade secret. Why would I want to reveal my subcontractor?

I emailed NCPP President Evans Witt (who is also the CEO of Princeton Survey Research Associates International - PSRAI) for comment. This is his response:

One of the issues that we deal with at NCPP is that the question of "Who did the poll?" or "Who was the pollster?" can be answered in a surprising number of ways. It arises more clearly for NCPP since NCPP is an organization whose members are organizations.

For example, The New York Times poll is simple: they designed and conducted the poll and did the interviewing in-house.

ABC News, for example, used ICR for a recent poll, providing this disclosure: "Sampling, data collection and tabulation by Social Science Research Solutions at ICR-International Communications Research of Media, Pa."** Very good disclosure, even if it is not necessarily simple.

PSRAI does for polls for Newsweek, and (usually) the telephone interviewing is done at Braun Research in Princeton, N.J., (a company independent of PSRAI) under PSRAI supervision.

Or take a survey that is done for an interest group via a public relations firm. The public relations firm may or may not have internal survey expertise. So the PR firm may have contracted out just the telephone interviewing or the PR firm may have contracted out the design, execution and interviewing. All are possible and legitimate arrangements.

To try to cover all this complexity, the field house requirement was included in the standard. Therefore, the standard covers the circumstance where the real knowledge about how the poll was done resides with the field house. It also helps to deal with a situation where there is a basic question of whether any poll was actually done, by providing at least one more avenue for asking for confirmation of the interviewing.

The discussion also raises some practical issues for us in trying to assess this aspect of disclosure. How do we know if an organization has fully disclosed field house, "where applicable?"

It is relatively easy for us to take note of field house when disclosed, as in the examples above. But how do we know when the need to disclose the field house "applies?" Consider that even very detailed New York Times "How the Poll Was Conducted" sidebar neglects to say explicitly that Times/CBS poll conducts interviews "in-house." If we flunk organizations for failing to make clear that they did not use a field house, my guess is that we'd strike a fair number of NCPP members.

But practical issues aside, I want to come back to the question that provoked this post from the news consumers' perspective. Does knowing the name of the field work provider, or knowing that pollster did interviewing in-house, help you assess the quality of the poll? Should this information be something we insist that pollsters disclose more explicitly?

*Correction*: We modified the response from Evans Witt at his request. Although ABC News used ICR for a recent survey, they more typically work with TNS of Horsham, Pa.

Comments

Researchrants.wordpress.com:

I don't know that it's particularly useful information to anyone outside the polling industry, and certainly not to your everyday news consumer or even your every day news reporter. But I question what "in-house" really tells us. In-house by whom, exactly? By interns? By professional interviewers who work full time for the news organization? By telemarketers from the subscription department?

Even knowing that interviews were conducted by XYZ Corporation of Tulsa, Oklahoma tells me nothing, including whether the calls actually originated from Oklahoma, or if XYZ outsourced offshore.

I think what WOULD be useful in this area, though, would be some basic descriptors of the field work -- dates (and times) of interviews, whether the interviewers were professional or students, whether they were under trained by the polling organization or by the field house, who supervised them, and so on.

It is certainly true that if I saw a new poll today and was told that the field work was done by XYZ Corporation of Tulsa Oklahoma I would discard the field-work information other than to give a minor plus to the poll because the pollster is not hiding the identity of the field work provider.

However, pollster.com is, among other things, a data bank of a large number of polls. Knowing the field work provider for each poll strengthens the data bank considerably. If, for example, a poll shows signs of poor field work it will be possible to pull up polls with the same field work provider to see if there is a general pattern. So over time this information could turn out to be highly useful.

A related point is that field work providers will undoubtedly be influenced by knowledge of whether or not their identities will be disclosed. It is much easier to be sloppy if you know that will not have to publicly take responsibility for your work. Knowing that your name will be on the final product is an incentive to do good work.

Maybe I'm missing something but I have trouble getting my head around the trade secrets argument in this context. Is it that firms are hoping to conceal the identities of their field work providers so they cannot be hired by competitors? Wouldn't field work providers try to advertise their existence? I don't get it.

Post a comment

Name:

Email Address:

URL:

Comments: (you may use HTML tags for style)

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.