News headlines aside, it’s important to understand that our result shows a competitive election, not a statistically meaningful advantage for either Love or Owens.
This post was written by Chris Karpowitz and Jeremy Pope, Co-Directors of the Center for the Study of Elections and Democracy. Inquires should be directed toward the authors.
Our 4th District Utah Voter Poll results received a great deal of media attention yesterday, and that attention raised some important questions about the nature of our survey and the best way to characterize the results. We followed the same process in the October UVP as we have for many previous surveys, and we have confidence in our methodology, which has yielded a reliable measure of public opinion in the state for many years. We always report the results as we receive them, regardless of the candidates, parties, or issues. Still, because candidates and parties tend to want to spin the results to fit their interests, it is important for political observers to get beyond the headlines to understand both the strengths and the limitations of this particular set of results.
As we emphasized yesterday, ours was a statewide sample of voters who were randomly selected to participate in the Utah Colleges Exit Poll and also agreed to be part of our Utah Voter Poll panel. We do not have a separate sample for each of the state’s four congressional districts. Among other things, this means that the number of respondents in each congressional district is smaller than the full sample and, thus, that the margin of error for the congressional district results is higher than the uncertainty associated with the full sample. For a simple random sample, the margin for a survey similar in size to ours would be plus or minus 3.4%. The analogous margin of error for the 4th District questions would be plus or minus about 6.4%. Clearly, neither candidate exceeded this margin of error, which means that neither candidate had a statistically robust lead at the time the survey was in the field.
Every survey includes some level of uncertainty (and different kinds of uncertainty at that), and as we tried to emphasize yesterday, our results are best interpreted not as a “lead” for Doug Owens but as a race in which neither candidate has substantially distanced himself or herself from the other. News headlines aside, it’s important to understand that our result shows a competitive election, not a statistically meaningful advantage for either Love or Owens.
So what is going on with our results? We see several possibilities, a few of which we will detail below.
One simple explanation is that the race was genuinely tight during the week we were in the field. According to press reports yesterday, the Love campaign said that they noticed similar tightening in their internal tracking polls earlier in the month, but according to their description, the gap between the candidates has widened again since the time our poll was in the field. We only have data from October 15-22, so any changes since that time, no matter what the partisan direction, would not be captured in our results. We simply cannot say what has occurred in the days since our survey closed.
Another possibility is that our sample leans more heavily Democratic than the electorate is likely to be on Election Day in 2014. Of course, the final resolution of that possibility won’t be known until next week. We noted in our post yesterday that our sample looked relatively similar to the 2012 4th District electorate. In 2012, the Utah Colleges Exit Poll found that about 29% of voters self-identified as Democrats and 51% identified as Republicans, with the rest being independents, identified with third parties, or identifying with no party at all. In our October 2014 UVP, about 28% of 4th District voters self-identified as Democrats and 55% identified as Republicans.
But midterm elections are not the same as presidential elections, and given the fundamentals of the election, with Democrats burdened by a relatively unpopular president in the 6th year of his term, it’s possible that the electorate will look quite different in 2014. What difference would this make to our results? To answer that question, let’s assume that partisans divide their votes in the way our survey showed, with Owens wining almost all of the Democratic votes, a plurality of the independent vote, and about 22.5% of Republican votes. Assuming Owens performs that way, what happens when we simply alter the mix of Republicans and Democrats in the electorate (holding constant the proportion of independents)? The table below presents different scenarios.
| 55% R 28% D |
57% R 25% D |
62% R 20% D |
64% R 18% D |
|
| Owens | 45.8% | 43% | 40% | 38% |
| Love | 42.2% | 44% | 47% | 48% |
In other words, if we assume everything else about our survey is correct and simply vary the partisan mix of the electorate, a statistical tie quickly becomes a substantial victory for Love.
One additional, related possibility involves examining the data with an even finer-tooth comb. In our survey, we asked voters to identify themselves as either strong partisans, not-so-strong partisans, or independents who nonetheless lean in a partisan direction when asked to choose. More respondents in our sample described themselves as Republican “leaners” than strong Republicans. But it is also possible that on Election Day, that mix will be different and more voters will describe themselves as strong Republicans than as weak or leaning Republicans. This may be especially likely in a midterm election, when strong partisans are often over-represented relative to presidential election years. Why would that matter? Because in our sample Love wins 90% of the strong Republican vote, but only around 60% of the weak and leaning Republican vote. Holding the total percentage of Republicans in the sample constant but altering the mix of leaning and strong Republicans would give Love a small lead in our survey. In other words, if a greater percentage of Republican voters on Election Day are the strongest partisans, Love is likely to significantly outpace our survey’s estimate.
The alternative scenarios we outlined above assume that Owens wins a little over 20% of the Republican vote, as he did in our survey. Winning a healthy percentage of Republicans is necessary for Democrats in a district that leans Republican. Falling short of that standard would make his path to victory next to impossible. To win, Owens must replicate the Matheson pattern of winning essentially all the Democratic, most of the independent, and more than 20% of Republican votes. One important trend to watch will be how Owens performs among self-identified Republicans, and the Utah Colleges Exit Poll will be a helpful tool to answer that question on Election Day.
In our survey, we also found that third-party candidates did relatively well, taking about 5 percent of the vote overall. The Libertarian candidate, Jim L. Vein, made an especially good showing, with 3.6% of voters preferring him. Perhaps, though, voters would make different choices if they knew the race was close. None of the third-party candidates is likely to win this time around, and in a competitive election, voters who expressed a preference for the Libertarian, Constitution, or other candidates on our survey might swing the outcome by choosing one of the two major party candidates when they get to the voting booth.
Finally, we want to emphasize again that every survey is only a snapshot in time and every survey estimate is accompanied by uncertainty. All polls should simply be treated as indicators of the true, underlying state of the race at any given time. No poll should be read in isolation.
Given these basic facts, our mid-October results do not show either candidate with an insurmountable advantage among voters who say they have made up their minds about how to vote. Nonetheless, Love enters the last week of the campaign with other sorts of advantages, not the least of which is fundraising. We expect that her fundraising resources will play a key role in the final days of campaigning. Love will likely be able to bring advertising and get-out-the-vote efforts to the race that Owens probably cannot match. Still, as we concluded yesterday, the final sprint to the finish is worth watching closely.
It was extremely irresponsible to release this poll in the manner that it was as it may have improperly influenced the race. You guys should issue a press release that your poll did not have sufficient data to make the claim you did. Answers were self-selected providing no relevant evidence of true scientific methodology. I graduated from BYU in Poli Sci and recommend a retraction in the spirit of fairness.
Your recommendation is based on two false premises. First, respondents did not self-select into the panel. They were recruited into it through random selection, as is the case of good public opinion polls. Second, there was indeed sufficient data for the claim presented. There were enough respondents in CD4 to allow for a margin of around 6 percentage points. Increasing the sample size to around 1000 in CD4 would have reduced that margin modestly to 3-4 percentage points. That’s an improvement, but not a huge one.
I was not directly involved in drawing the sample used here, and somebody who was might chime in. But remember that those criticizing this poll (making roughly the claims you seem to be echoing) have political reasons to do so.
Prediction for you based on my analysis of the data = Mia Love 49-Owens 42.
You guys can give me an honorary doctorate if I prove correct.