Why pollsters should release their topline results

Polls can be done well or poorly. Releasing topline results aids the public in detecting poor polls.

The Tribune reported a Mason-Dixon poll last week suggesting a wide Republican advantage in the race for Salt Lake County mayor. Yesterday, the Tribune published a new story with “revised” results showing a tighter race. The initial error might have been avoided–or at least caught more quickly–if Mason-Dixon had been more transparent by releasing so-called “topline” results.

Polls can be done well or done poorly. When done well, a poll can provide a reasonable estimate of public opinion (taking account, of course, of the poll’s margin of error). There are three main ways that a poll can go wrong.

First, sampling error. If the pollster doesn’t contact a representative sample of voters, the poll will be off. That was apparently the problem with this Mason-Dixon poll. Their sample included a disproportionately large number of Republicans.

Second, non-response error. Not everybody contacted by the pollster will participate. If those who decline are systematically different from those who participate–for example, if more Republicans agree to take the poll than Democrats–that will also introduce error.

Third, measurement error. If questions are worded poorly, then you’ll get misleading results. Reputable pollsters are very careful to avoid measurement error. For example, the best polls will randomly rotate the order in which options appear. So, when asking about preferences in a presidential vote, some respondents will hear Obama’s name as the first option, while others will hear Romney’s name first. There are many ways to minimize measurement error, and reputable pollsters are mindful of them.

To aid the public in detecting faulty polls quickly, pollsters should transparently disclose their sampling method, their response rate, and their questionnaire. This disclosure is sometimes referred to as “topline results.” Topline results also include the pattern of responses to each question. If Mason-Dixon had released their topline results with the initial poll, readers might quickly have noticed that the mix of Republican and Democratic respondents seemed off.

(Incidentally, readers should always be suspicious when campaigns release their internal poll numbers, since those releases almost never include full topline results.)

If you search this site for “topline,” you’ll find that those who post poll results here (usually Quin Monson and Kelly Patterson) routinely post their topline results. At a minimum, when we write about polls, we include a methodological note at the bottom giving some information about the polling methodology for the benefit of polling nerds.

Polls can be done well or poorly. Releasing topline results aids the public in detecting poor polls.

Possibly related posts:

About Adam Brown

Adam Brown is an assistant professor of political science at Brigham Young University and a research fellow with the Center for the Study of Elections and Democracy. You can learn more about him at his website.
This entry was posted in Everything and tagged , , , . Bookmark the permalink.