This post was jointly written by Quin Monson and Kelly Patterson.
Recently, the Libertas Institute posted a “survey” on their web site purporting to measure how Utah voters feel about a possible statewide anti-discrimination law to protect gay and transgender Utahns against housing and employment discrimination.
The survey has many issues that cast serious doubt on the validity of the results. Here is a brief summary of some of the problems.
1. The questions are unbalanced and inaccurate. For example, the two questions about an anti-discrimination law contain references to “jail time.” However, jail time is not part of anyone’s anti-dicrimination ordinance or law. What Libertas calls “contextualized” simply means asking loaded questions by introducing the threat of a non-existent punishment.
In addition to inaccurate questions about anti-discrimination, the question order is a serious concern. The relevant questions are preceded by questions about free association and religion. This order effectively biases the survey participants by having them consider reasons why they would not want a law right before they are asked about a possible law.
2. The robocall methodology used by Libertas is not methodologically sound. Robocalls cannot obtain a valid representative sample of any population of interest because they have no way to select or ask for a respondent within the household they are calling. Many valid surveys ask for a respondent by name from a list. Or, if the poll is conducted on a random list of phone numbers, a method is employed to select a person at random within the household. For example, many pollsters ask to speak to the person who celebrated the most recent birthday. This poll could have been answered by someone’s 14 year old, and they would never know. The survey is supposed to represent voters, but this is virtually impossible to verify. The poll contains no gender question. Because they take whomever answers the phone, it is likely that the respondents are disproportionately female. Voters in Utah are evenly split 50/50 between men and women. Finally, a heavy proportion (37%) of the respondents are over 60 years old. This too is disproportionately high.
3. The survey also has some serious ethical and possible legal problems. One serious issue is that they posted a data file on their web site that includes the actual phone numbers of the people they called. This is a huge violation of trust. The American Association of Public Opinion Research follows a code of professional ethics that you can find here. The relevant part that this survey violates says, “Unless the respondent explicitly waives confidentiality for specified uses, we shall hold as privileged and confidential all information that could be used, alone or in combination with other reasonably available information, to identify a respondent with his or her responses. We also shall not disclose or use the names of respondents or any other personally-identifying information for non-research purposes unless the respondents grant us permission to do so.” They did not release the full script of the robocall, but we are guessing that they did not tell participants that they would be releasing their phone numbers publicly. This would have led many people to hang up on the call and would make many of the existing respondents very angry.
We are also worried that they robocalled cell phones. If so, this is actually illegal under federal law. They could be in serious trouble if someone filed a complaint.
In their write up of the “survey,” Libertas claims that “These results validate our initial theory…” Unfortunately, good social science does not validate theories, it simply rejects null hypotheses at certain levels of confidence. We have no confidence in this survey.
My limited background in marketing research led me to believe this poll was not as accurate or methodologically sound. Thank you for filling in the gaps and exposing this as a flimsy and poorly done poll.
One problem with your critique: towards the end you suggest the survey may be illegal without providing evidence of this claim beyond your personal “worry.” That is a pretty hefty implication to throw out without backing it up with some form of substantial evidence.
Suggesting that someone’s actions are illegal or criminal without providing evidence seems at least as unethical and unprofessional as conducting a methodologically unsound poll.
Derek Monson
We only worry because there is no information on how the poll was conducted. We did not suggest that they were illegal, only that they might be. It is up to the sponsor of the poll to provide as much information as possible about the methodology. In the absence of that information, it seems appropriate to “worry” about implications of that particular methodology. Would it be better for the public good to bury our concerns in the absence of any information about how the poll was actually conducted?
Our assertion that the survey may have done something illegal was based on the possibility that the poll used an “automated phone system” to dial cell phone numbers. I suppose that this could be verified using the full data file that Libertas originally made available for download before realizing that it contained the full phone numbers of the survey respondents.
Our suggestion that the poll may have engaged in something illegal prompted a response from Libertas that said, in part, “… it appeared that we were incorrectly informed that non-profits were exempt from certain FCC restrictions, but yesterday’s survey prompted additional research into the regulations, including calls made directly to the FCC, to discover that this was not true. As such, we will be suspending our polling activities for the foreseeable future while we take the time to better determine the relevant regulations…” It looks like we were right on this point.
The current version of the data file that Libertas has released includes the last four digits of the phone number. I’m pretty sure that a determined individual could use these last four digits of the phone number and the reported age in the survey together with the state voter file and still identify many of the respondents to this poll. I applaud the effort at transparency, which after all is largely what enabled this original critique, but potentially identifying information has to be deleted from the file before releasing it publicly.
Interesting that this “non-partisan” report would choose to critique the Libertas poll but not the original poll. The original poll for Equality Utah has it’s own misgivings and biases, yet this body finds no reason to write a report about it? That would seem pretty partisan to me.
I’m not sure if I know which poll you’re referring to. Is it this one from December 2011? http://www.ksl.com/?sid=18526785. Here is a link to the full results: http://www.equalityutah.org/images/stories/PDFs/Read_the_full_Survey_with_highlights_HERE.pdf. I don’t see huge problems with this survey. The main question of interest, whether Utahns favor or oppose and anti-discrimination law, is the fourth question in the survey. The first question is a simple right track/wrong track question about Utah in general. The next two questions are about knowledge of the current law, something you’d have to ask before asking if someone favors or opposed a proposed law. You could quibble with how some of the later items are worded, I suppose, but I don’t find a lot of problems with the main claim of the survey.
The writer claims that “jail time” isn’t mentioned in the law,
ignorantly (at best) failing to grasp the perspective of Libertas.
Like the classic quote “the more laws, the less justice” — Cicero
libertarians commonly follow the logical trail of all laws…
if you don’t comply, you will be cited, if you don’t pay the fine
you will be further penalized, if you refuse penalty you will be jailed,
if you refuse to be taken to jail you will be physically coerced,
if you resist physical coercion you will be shot/killed.
If that is truly the line of reasoning, perhaps the survey question should have said that the punishment was execution instead of jail time.
I don’t see how what you wrote can be considered anything other than suggesting that it was illegal.
While I’m at it, here are my suggestions to your three points.
1. Jail time: As far as I’m aware if you don’t pay your fines you do go to jail, so it is in every proposed anti-discrimination law. Regardless of jail time, according to Chief Justice John Marshall “the power to tax involves the power to destroy”. They could have used the word destroy rather than jail time. Sounds like choosing to refer to “fines or jail time” was moderate, anything less would have been inaccurate.
2. The accuracy of a Robocall is proportional to the size of said survey. The gender bias on the other hand remains, there are countless variables and not all can be accounted for. Income bias is rarely accounted for in surveys and in many regards influences opinion more than gender. The only perfect survey would include every potential voter, that being impossible a error margin is acceptable.
3. A silly mistake although it does not effect survey accuracy.
My comment was supposed to be a reply to the authors comment, apparently i clicked somewhere else after. o well.