Why does Media Matters prefer the single survey presented in aAmerican Journal of Preventative Medicine (AJPM) article to the 23 NACOP and other surveys? Let us compare the AJPM article and the NACOP surveys since they are the ones that both deal with command officers. Both were conducted through the mail, but each year, for 23 years, the NACOP survey was sent to all of the roughly 22,000 chiefs of police and sheriffs in the United States. By contrast, the AJPM article was only done one time during 2002/2003 and they sent out a letter to just a small fraction of these police chiefs, merely 574, or fewer than 3 percent of the total.

In addition -- something Media Matters conveniently neglects to mention -- the researchers for the AJPM article purposely did not examine a representative sample of police chiefs, limiting themselves to only a subsample of those cities with more than 25,000 people.

One difference between the NACOP and the AJPM article surveys is that the latter randomizes who they sent the survey to, something that Media Matters seems to think is an advantage of the AJPM article survey. But that just shows how little Media Matters understands about statistics. "Randomization" is a method researchers are typically forced to use because they only have limited resources to contact a smaller number of people. Randomization is always imperfect to some degree in that it never perfectly represents the entire population; it is merely an an attempt to get a sample that is reasonably representative of the entire population.The NACOP did not have send out a mailing to a subset of police chiefs and thus did not have to randomize because they sent the survey to the entire set of chiefs of police and sheriffs in the US. That is of course vastly better.

The response rate for the NACOP surveys was between 12% and 14%, thus ranging from about 2,700 to 3,150 per year. By contrast, the AJPM article had responses from 405 chiefs. Although the smaller survey had a better response rate, it represents a very small number of the approximately 22,000 police chiefs and sheriffs across the country. The NACOP survey had more than 7 times the number of respondents in any single year that the AJPM article had. And it has been repeated 23 years, including recent years.

As Media Matters correctly points out, the answers from the NACOP survey are not weighted by the population characteristics. However, Media Matters neglects to mention that the survey in the AJPM article doesn't do that either. Media Matters attacks the Police Magazine survey for not carefully weighting for "the demographics of its survey population to be representative of the general population," but they don't hold the survey in the AJPM article to the same standard. Indeed, it is clear from the AJPM article that they heavily oversample urban areas (62.4%) in comparison to suburbs (36.3%) or rural areas (1.2%).

This huge difference in the number of responses collected can easily explain the differences in the results. The 2010 NACOP survey shows that 77% of chiefs of police and sheriffs supports very liberal concealed carry laws. Twenty-three percent of respondents who either oppose or are neutral on concealed carry implies a number that is larger than the entire number surveyed for the AJPM article, let alone the 236 who say that civilians should be "prohibited from carrying a firearm in public places."

There are also problems with the questions used by the AJPM article. Two of the four the questions from the AJPM article that are referenced by Media Matters are extremely vague and are likely to elicit agreement simply because of their vagueness. For instance, agreeing that"Civilians [should be] prohibited from carrying a firearm in public places," cannot reasonably be interpreted as opposing concealed carry by law-abiding citizens who pass a criminal background check. If an urban police chief could think of one place that they would oppose people openly carrying guns in public places, they would agree with this statement.

The "child access laws" question pointed to by Media Matters is equally vague. The question could mean anything from a minimum age requirement for purchase of a gun to whether a child can only use a gun in the presence of an adult to prohibiting those under 18 access to guns under any circumstances.

Media Matters attacks the NACOP for not making information of their survey available, but Barry Shephard, the NACOP's Executive Director, wrote me: "We do not have record of any inquiry or request for information at any point in time from Media Matters."

Click on pictures below to make text larger. Because Media Matters has rewritten their comments in the past, I am providing a screen shot here of what they wrote.

2 Comments:

"Why does Media Matters prefer the single survey presented in a American Journal of Preventative Medicine (AJPM) article to the 23 NACOP and other surveys?"

Maybe I missed something, but I thought the Media matters post was taking issue with the ONE survey by the NACOP that you quoted in the WSJ. Your question that I quoted above is a bit strange, don't you think? What other surveys?

I just reread their post and your rebuttal and I have to say I found theirs very convincing and yours a bit weak and confusing. You seem to be saying "my survey is better than your survey." While the Media Matters article presented several problems with the NACOP itself.

Dear Mike B:The NACOP has done this survey over 23 years, and gotten very consistent results. The fact that Media Matters wants to acknowledge only one of those surveys seems irrelevant.

Again, your response here isn't very useful. You might think saying it is just one side versus the other is something important, but it ignores all the specific points that I raised. For example, the survey that Media Matters likes isn't even a survey of all police chiefs. They should say that police chiefs of larger urban areas may hold a particular viewpoint, but it doesn't apply to all police chiefs. You also don't respond to these questions being relative vague, so it isn't even clear what the questions actually are measuring.

Surveys with seven times the sample size and redone over many years and clearer question might also be given more weight than a one single shot survey with vague questions. Take a simple point: why doesn't sample size matter?

Obviously I made other points, but just saying my response is weak without actually pointing out why you think that is the case isn't very useful.