This post was jointly written by Quin Monson and Kelly Patterson.
Recently, the Libertas Institute posted a “survey” on their web site purporting to measure how Utah voters feel about a possible statewide anti-discrimination law to protect gay and transgender Utahns against housing and employment discrimination.
The survey has many issues that cast serious doubt on the validity of the results. Here is a brief summary of some of the problems.
1. The questions are unbalanced and inaccurate. For example, the two questions about an anti-discrimination law contain references to “jail time.” However, jail time is not part of anyone’s anti-dicrimination ordinance or law. What Libertas calls “contextualized” simply means asking loaded questions by introducing the threat of a non-existent punishment.
In addition to inaccurate questions about anti-discrimination, the question order is a serious concern. The relevant questions are preceded by questions about free association and religion. This order effectively biases the survey participants by having them consider reasons why they would not want a law right before they are asked about a possible law.
2. The robocall methodology used by Libertas is not methodologically sound. Robocalls cannot obtain a valid representative sample of any population of interest because they have no way to select or ask for a respondent within the household they are calling. Many valid surveys ask for a respondent by name from a list. Or, if the poll is conducted on a random list of phone numbers, a method is employed to select a person at random within the household. For example, many pollsters ask to speak to the person who celebrated the most recent birthday. This poll could have been answered by someone’s 14 year old, and they would never know. The survey is supposed to represent voters, but this is virtually impossible to verify. The poll contains no gender question. Because they take whomever answers the phone, it is likely that the respondents are disproportionately female. Voters in Utah are evenly split 50/50 between men and women. Finally, a heavy proportion (37%) of the respondents are over 60 years old. This too is disproportionately high.
3. The survey also has some serious ethical and possible legal problems. One serious issue is that they posted a data file on their web site that includes the actual phone numbers of the people they called. This is a huge violation of trust. The American Association of Public Opinion Research follows a code of professional ethics that you can find here. The relevant part that this survey violates says, “Unless the respondent explicitly waives confidentiality for specified uses, we shall hold as privileged and confidential all information that could be used, alone or in combination with other reasonably available information, to identify a respondent with his or her responses. We also shall not disclose or use the names of respondents or any other personally-identifying information for non-research purposes unless the respondents grant us permission to do so.” They did not release the full script of the robocall, but we are guessing that they did not tell participants that they would be releasing their phone numbers publicly. This would have led many people to hang up on the call and would make many of the existing respondents very angry.
We are also worried that they robocalled cell phones. If so, this is actually illegal under federal law. They could be in serious trouble if someone filed a complaint.
In their write up of the “survey,” Libertas claims that “These results validate our initial theory…” Unfortunately, good social science does not validate theories, it simply rejects null hypotheses at certain levels of confidence. We have no confidence in this survey.