Polls And Voteshttp://pollsandvotes.com/PaV
Polls in Perspective, Votes in ContextMon, 05 May 2014 15:34:41 +0000en-UShourly1http://wordpress.org/?v=3.9.14Survey Mode, Population and the Party ID Distributionhttp://pollsandvotes.com/PaV/2014/05/survey-mode-population-and-the-party-id-distribution/?utm_source=rss&utm_medium=rss&utm_campaign=survey-mode-population-and-the-party-id-distribution
http://pollsandvotes.com/PaV/2014/05/survey-mode-population-and-the-party-id-distribution/#commentsMon, 05 May 2014 15:34:41 +0000http://pollsandvotes.com/PaV/?p=586Continue reading →]]>It is widely understood that the population sampled, whether adults, registered or likely voters, has some effect on the distribution of partisanship. But mode differences among live interview, automated phone and internet data collection also shift the distributions. Pew polls are shown as a high quality live interview reference compared to all live interview polls.

The most prominent effect is on the size of independents. Both automated phone and internet modes produce substantially smaller estimates of Independents than do live interview phone polls (which include cell phone strata.)

Independents are on average 34.8% of live phone samples but 28.0% and 27.6% of automated phone and internet samples respectively. This is in line with what we might expect to find if the smaller response rate to automated polls and the “volunteer” samples of internet polls are correlated with political interest and likely with partisan intensity.

Where the independents go in the different modes varies a bit. Republicans are substantially more numerous in automated samples (33.8% vs 26.7% in live phone) but not so in internet samples (25.8%). Meanwhile Democrats are more common in both automated phone (38.1% vs 34.0% for live phone) and (more modestly) for internet (35.1%).

One might suspect that mode and population sampled are highly correlated and that this accounts for some of the differences. Happily, there is enough variation in population within mode to estimate the effects of both mode and population.

The estimates below are from a linear regression of the partisan make up of each sample on mode and population with a polynomial in time to capture trend effects (not shown.)

The data are from HuffPost/Pollster’s collection of 982 polls that reported their partisan make up between September 3, 2008 and April 27, 2014. (Two “mixed mode” polls were deleted. No account has been taken of changing methodologies, such as reported blending of automated phone with internet samples, the detailed timing of which is not available.)

Estimated effects are given below. All are statistically significant except the Rep/Internet coefficient.

While both partisan camps gain when we move away from the live interview mode, there is a modest pro-Republican advantage with larger gains. Independents shrink in each case.

While partisan bias is an obvious potential concern, the higher involvement and less independent nature of automated modes may also have more subtle effects, from over stating turnout and candidate awareness to possibly reducing the apparent impact of dynamic factors as more partisan samples are less likely to be moved by events or advertising.

A final note is to compare the distribution of each partisan category between the Pew polls alone and all live interview polls. The considerably greater spread of the distribution for all live polls is a vivid example of house and question wording effects. The very wide spread of the independent distribution in particular is notable. We may assume there is a clear distribution of partisanship (as in the trends below) but the spread around these trends is quite large. Certainty as to the sizes of each partisan camp is illusory. Central tendency should not ignore spread.

An update as of June 3 is added at the bottom. Still no change in approval trend, with Quinnipiac poll included.

A lot has been written about the impact, or rather the lack of impact, of the recent Benghazi hearing, IRS and AP phone records revelations on President Obama’s job approval. I was sitting this one out until Steve Kraske of the Kansas City Star called Thursday to talk about the issue. His piece appeared today and is nicely done. I ran an analysis of my own in order to answer his questions and that analysis is what you see here.

The chart above shows Obama approval since January 1, 2013 using all 162 polls taken through May 23. The red line is the estimated trend using a simple quadratic model, the smoothest trend for these data. That trend put approval at 53.5% on January 1, and at 48.9% on May 23, a decline of 4.6 points over the year so far. But as the red line shows, that decline has recently flattened and little has changed since the end of March. The red trend estimate was 48.9, as it is now, on March 26, and declined to a low of 48.5 on April 23, after which it has returned to 48.9. None of those recent changes are statistically significant.

The blue line in the chart is my standard local regression measure of approval trend, set to a moderately sensitive level. If the blue line departs from the red line, that suggests some time points where the local trend is different from the longer term smooth trend, possibly suggesting some interesting political effects. As it happens, the blue line has tracked quite closely with the red smooth trend, and especially so in the last month where the lines are all but indistinguishable.

An alternative approach compares the 16 polls taken in the 14 days before the May 8 hearings to the 17 polls taken in the 14 days after. Those data are shown in the chart below, with the means and distributions of the before and after data. The pre-hearing approval mean is 48.69 while the post hearing mean is 48.71. Not much difference.

Given these charts, it is no surprise that a statistical model finds no evidence at all of a change in trend or mean since the Benghazi hearing on May 8 and the subsequent IRS and AP revelations. That conclusion fits well with virtually all the other analyses on this topic.

It is possible, however, that the overall trend masks some shifts among partisan groups. Democrats and Republicans are apt to react differently with shifts among independents the most important indicator. We have, thanks to Pollster.com, 43 polls since January that break out approval by partisanship. That gives us only limited information about change since the Benghazi hearing, but any such shifts may at least be a canary in the coal mine. The statistical analysis of these data show nothing close to statistically significant, so the shifts visible in the charts below, and the estimated effects, should be treated with great skepticism. The test will come over the next month or two as data accumulate and any “real” impact will prove enduring or will wither away to nothing.

The three charts below plot the red overall trend line for each partisan group, and then overlays a relatively sensitive blue local regression line. Since the hearings on May 8, there has been a slight uptick in approval among Democrats and a slight downturn among Republicans. The statistical model estimates Dems are up 2.8 points and Reps down 3.2 points, though neither are close to statistically significant.

Among independents, the move is, as with Democrats, upward since May 8, by 2.7 points, still not statistically significant.

So the wise statistician would say “no evidence” of change among any of these groups. I would agree the evidence is far too weak to reach any conclusion of an impact yet. But if you read tea leaves and stare into crystal balls there is the slightest suggestion that independents are not being driven away from Obama at this point.

In the rush to find instant effects of events, we look at data before there is enough evidence. But more important, the enduring political impact of events, if there are any, are not usually things we see in a week or even two but rather shifts in trends that set approval on a new trajectory, whether up or down. Rare events, such as the killing of bin Laden do produce almost immediate jumps in approval, but those events usually prove fleeting and the trend rapidly returns to its previous track. The real test of the impact of events over the past two weeks will come in six months when we can look back and see if May 8 represented a turning point after which the trend in approval shifted, either up or down, with lasting consequences. Such enduring shifts are the events that reshape presidencies, not short term bounces.

June 3 Update:

As more data are available it is undoubtedly true that there will eventually be some movement from the current trend, either up or down. Note, though, that the longer such a shift is removed from the breaking news of the three scandals, the less certain we can be that those were the reason for the shift in trend. Further hearings or new findings may yet occur but so far we are not seeing significant shifts.

A summary of the statistical models finds no statistically significant shift since May 8, either overall or for any partisan group (Dem, Rep or Ind). The Gallup daily poll, taken alone, has a not-quite significant negative coefficient, estimated at -2.1 points on approval (p=.080) but the Rasmussen daily has a positive but not significant estimate of +0.93 points (p=.36).

Here are where the state polls stand relative to the national polls on election day morning. The trends are estimated using a common latent trend for this comparison. The national trend estimate is +0.4 for the Obama minus Romney margin.

From a presentation I did today at the Ethics and Elections Conference, sponsored by the Center for Journalism and Ethics at UW-Madison. #UWEthics if you are interested.

Gas prices are a fairly “objective” political fact. The chart above shows prices since 1990. This has been the subject of some comment, of course.

In our March Marquette University Law School Poll (which I direct) we asked “Is the price of gas something a president can do a lot about or is that beyond any president’s control?” In Wisconsin, opinion is evenly divided:

But Dems and Reps have sharply differing views:

Republicans are largely of the view that presidents can do a lot about gas prices, while Democrats are convinced market prices are beyond the power of presidents to control.

Or so it is in 2012. In May 2006, when the CBS News Poll asked this exact question as gas prices spiked during the Bush administration, the partisans had just the opposite theories of presidential control of the economy:

In 2006 a large percentage of Democrats were convinced presidents can affect gas prices. Republicans were less convinced (though split a little more evenly, 45-55, versus 75-25 for Dems).

The point of the talk is simply that our ability to perceive and interpret gas prices and presidential responsibility is the tool of our partisanship, not so much determined by our economic theories.