Utah’s abysmal turnout is not caused by bloated registration rolls. It is caused by citizens choosing not to participate.
Yesterday, a legislative committee debated legislation that would remove a citizen from the voter registration rolls if he or she fails to participate in consecutive elections. The bill’s sponsor claims that this legislation would boost Utah’s voter turnout rate compared to other states. After all, he reasons, turnout is calculated by dividing the number of ballots cast by the number of total registered voters, so removing inactive voters from the registration rolls will have the effect of increasing the turnout percentage.
This bill may have some usefulness when it comes to efficient administration of state elections. But let’s consider this argument about turnout for a moment.
As it happens, researchers do not calculate turnout as a percentage of total registered voters. They calculate it as a percentage of those eligible to vote, regardless of whether they are registered. “Eligible” means at least 18 years old, a citizen, and (in most states) not incarcerated. It is true that Utah’s government calculates turnout as a percentage of registered voters, but that is an unorthodox method, one I have written about previously.
But here’s the rub: Utah’s declining turnout isn’t caused by bloated registration rolls. Rather, it is caused by fewer eligible citizens voting. Maybe that’s because they aren’t registering in the first place; maybe that’s because they are registered but not showing up. But even when we measure turnout correctly (as a percentage of eligible voters), we see that Utah has fallen to the bottom of national rankings on voter turnout. Here’s the data (courtesy of Michael McDonald at George Mason University).1
Year
Utah’s turnout (% of eligible voters)
Utah’s turnout rank (% of eligible voters)
2010
36.2%
47
2008
56.0%
45
2006
34.3%
45
2004
58.9%
32
2002
37.8%
39
2000
53.8%
34
1998
35.8%
37
1996
50.2%
34
1994
41.5%
29
1992
64.0%
15
1990
40.7%
25
1988
62.0%
7
1986
42.5%
21
1984
63.0%
10
1982
55.1%
6
1980
66.0%
5
The trend is clear. From 1980 until 1992, Utah’s turnout was consistently in the top half of national rankings. From 1994 on, Utah’s turnout has been in the bottom half. Since 2006, Utah’s turnout has been in the bottom tenth. Utah’s abysmal turnout is not caused by bloated registration rolls. It is caused by citizens choosing not to participate. For whatever reasons, citizens aren’t registering to vote, and even if they are registered, they aren’t voting. Purging voters from the registration lists is not going to improve this problem.
Legislators may be showing an increased willingness to run legislation that has not been subject to a thorough vetting during the interim months.
The Utah legislature opened its 2012 general session today. From a quick count, it appears that 211 bills were introduced this morning. Introduction is the first step toward passage, although many of those bills will ultimately fail. Let’s compare the 211 bills introduced today to the first-day numbers from the past few years.
Year
Bills introduced on first day of session
Total bills introduced in session
Percent of bills introduced on first day
2007
292
733
39.8%
2008
328
744
44.1%
2009
276
800
34.5%
2010
226
713
31.7%
2011
191
782
24.4%
2012
211
–
–
Take a close look at the right-hand column. It tells you what percentage of bills were ready to go on day one of the session. From 2008 through 2011, there was a striking shift. In the first two years shown here (2007 and 2008), 40-44% of the session’s bills were introduced on day one. From 2009 through 2011 that percentage fell, until only 24.4% of bills were introduced on the first day of the 2011 session.
I’m only working with 5 data points here, so I don’t want to make too hasty a conclusion. Still, we might be witnessing an important shift. Legislators may be showing an increased willingness to run legislation that has not been subject to a thorough vetting during the interim months.
There are, of course, exceptions. Prior to the 2011 session, Stephen Sandstrom spent months vetting his immigration enforcement bill. Whether you liked his bill or not, you can’t say he didn’t spend lots of time working on it. Contrast that with certain other bills considered in 2011 (ahem) that passed with far less time for public scrutiny.
So what do we make of the 211 bills introduced today? Let’s assume that the total number of bills introduced this session will be somewhere between 713 and 800. These numbers seem reasonable if you look at the table above. In that case, then today’s 211 bills will represent between 26.4% and 30.0% of all bills that will be introduced in 2012. That’s better than what we saw in the 2011 session, but still worse than the 2007, 2008, 2009, and 2010 sessions.
An updated version of this post, written after the 2012 session concluded, can be found here.
Second, I’ve created legislator flashcards. You can learn the name, face, and position of all 104 legislators. I’ve been making these for legislative interns for a few years, but they are now public. Caveat: Due to recent resignations, some of the details in these flashcards will need to be updated in a couple of weeks. You can find Utah legislator flashcards here.
Media outlets would be wise to verify that they have the correct district maps.
I’m just now catching up on all the news from the past few days. It looks like people might be a little confused about district maps.
To illustrate a story about Jim Matheson’s chances in Utah’s new 4th Congressional district, KSL displayed the graphic below (at left). Trouble is, the graphic at right is the correct map (from this source). If you look at the 4th district (the small one in the middle), you’ll see that this map is very different from the one KSL displayed. There are also substantial differences if you look at the Uintah Basin or at southeastern Utah. (Click the image to enlarge it.)
KSL's incorrect map (left) compared to the correct map (right)
Just to make the difference obvious, I’ve zoomed in on the 4th district in the images below. (KSL’s story was focused on the 4th district.) You can see that KSL’s map is very incorrect. One of the more visible differences is that KSL’s map appears to stop north of US-6, whereas US-6 cuts across the middle of the official map:
Utah's 4th US House district according to KSL (left) and official maps (right)
Maybe this is an easy mistake to make. After all, legislators and activists floated a LOT of maps during the redistricting process. Of course, now that it’s been two months since the legislature adopted its final map (on October 17) and the governor signed it (on October 20), media outlets would be wise to verify that they have the correct district maps.
Gamblers are more willing to bet their money that Huntsman can beat Obama than that Romney can.
Intrade runs betting markets on just about everything. Because people are putting money on the line, the current intrade share prices can be used to estimate the “common wisdom” among bettors about how likely something is.
The political markets are fun to watch. There’s a market for betting on the outcome of the 2012 presidential election and another for betting on the Republican nomination race. At present, the betting on Intrade leads to these predictions (sorted in descending order by the middle column):
Candidate
Chance of being GOP nominee
Chance of winning 2012 election
Romney
68.5%
34.2%
Paul
8.4%
4.9%
Gingrich
8.2%
3.1%
Huntsman
6.7%
3.4%
Perry
3.0%
1.0%
We can take this one step further. If you were to bet on a candidate’s chance of winning the general election in 2012, you would have to factor in that candidate’s chance of winning the nomination first. By fiddling with these numbers, we can estimate each candidate’s electability in the general election.
For example, if we divide Romney’s 34.2% chance of winning the general election by his 68.5% chance of winning the GOP nomination, then we infer that Intrade’s bettors believe that Romney has a 49.9% chance of beating Obama if he receives the nomination. (That is, 34.2/68.5=49.9). If he is the nominee, bettors believe Romney has a 49.9% chance of beating Obama.
Let’s do that math for all these candidates. Here’s the result (in descending order):
Candidate
Math
Chance of beating Obama (if nominated)
Paul
4.9/8.4
58.3% chance
Huntsman
3.4/6.7
50.7% chance
Romney
34.2/68.5
49.9% chance
Gingrich
3.1/8.2
37.8% chance
Perry
1.0/3.0
33.3% chance
I question the result for Ron Paul. He has such an energized base that I wouldn’t be surprised if his supporters were irrationally optimistic about his odds. Hard to say.
But aside from Ron Paul, bettors believe that Huntsman is the most electable candidate. And in a result that may interest Utah readers, gamblers are more willing to bet their money that Huntsman can beat Obama than that Romney can. (Please, no jokes about $10,000 bets. I know. It’s tempting.)
I wrote a similar post back in June. Back then, Huntsman had a 48.0% chance of beating Obama, better than Romney’s 41.1%. Huntsman still has better odds against Obama, but Romney has closed the gap somewhat.
The 2010 exit poll suggests that 49.6%-52.0% of Utah Mormons are female. Meanwhile, the 2010 U.S. Census shows that 50.3% of voting-age Utahns are female.
After fruitful discussion with one of the authors of the report discussed here, I have made some changes to this post. I point out more significant updates with a brief “update” note, but I’ve made a few smaller changes without pointing them out.
I just read an article in the Tribune claiming that the gender gap is widening among Utah Mormons. The article is based on a recent report by two sociologists. It’s possible that the gap is widening, although another recent survey causes me to think twice.
For background, I’ll start by talking about what a margin of error is and how it should be interpreted. Then, I’ll present evidence from a separate survey that suggests a different conclusion.
What is a margin of error?
Every 10 years, the US government conducts a Census. They contact every person in the country and ask them several questions. Because they don’t use a sample, but attempt to talk to the entire population, they do not calculate a margin of error. They simply report the average age, income, gender, etc. of the entire American population.
But survey researchers don’t talk to the entire population. They talk to a sample of 500, 1000, or 2000 people. Because pollsters sample (unlike the Census), they cannot say exactly what a number is. They can only estimate. Using mathematical proofs, we know what the margin of error for any given sample will be. As your sample size grows, your margin of error shrinks.
Here’s the statistics from the report, as reported in the Tribune:
In 1990, a survey found that 52.5% of Utah Mormons were female.
In 2008, a survey of 270 Utah Mormons found that 60% of Utah Mormons are female.
“Mormonism, like most Christian denominations in the United States, has a surplus of women. In 1990, this surplus was more pronounced among Mormons outside Utah, where 54.9% of Latter-day Saints were female, compared to 52.5% in Utah. By 2008, a dramatic shift had occurred. While the male to female ratio actually narrowed somewhat in most of the nation, it widened significantly in Utah. Females now outnumber males in Utah 3 to 2 [among Mormons].”
The report then goes on to try to explain the “dramatic shift.” The trouble is, we first need to be sure that the shift from 52.5% to 60.0% was large enough to get out of the margin of error.
(Update: This paragraph revised.) The report itself doesn’t give the margin of error for both numbers, so I did my best to estimate them. With a sample size of 270, the margin of error on the 2008 poll would approach 6%. (For fun, you can use this link to calculate a margin of error for any sample size). The report didn’t specify the sample size from 1990, but based on other numbers in the report, I estimated it at around 600.1 That would produce a margin of error of 4%. Thus, it may be that the 1990 survey estimated that 48.5-56.5% of Utah Mormons are female, while the 2008 survey estimated that 54%-66% of Utah Mormons are female. If the margins of error overlap like that, it’s possible that the true percentage has been 55% or so all along.
(Update: This paragraph added.) After corresponding with one of the report’s authors, I am less concerned about this problem. He shared with me some additional details of their analysis. In statistical jargon, it does appear that the difference is “statistically significant” between 1990 and 2008. That is, after taking account of the actual sample sizes and margins of error, it does appear that the authors are justified in saying that the gender gap in 2008 was larger than in 1990, at least based on the surveys they use.
After corresponding with the authors, I no longer question the authors’ interpretation of the 1990 and 2008 statistical data. Their interpretation was sound.
Evidence from another data source
Even in a perfectly orthodox statistical analysis, and even with a perfectly designed survey, we know from statistical theory that we’ll have an incorrect result in 1 out of every 20 tests. That’s just the nature of social science research. It’s true of my own research, and it’s true of this report. The trouble is, we have no way of knowing whether one particular analysis is one of those false positives.
The best way to detect a potential false positive is through replication. (Of course, if two surveys disagree, we still don’t know which one is the “1 in 20” error, so the more replication, the better.)
So let’s look at another survey and see what we find. The 2010 Utah Colleges Exit Poll, a statewide sample of voters, found that 50.8% of Utah Mormons are female. The exit poll reached 6,717 Utah Mormons, a very large sample, resulting in a small margin of error (1.2 percentage points). Thus, the exit poll estimates that the percentage of Utah Mormons that is female is somewhere between 49.6% and 52.0%. That sounds pretty close to the 52.5% reported back in 1990.
For statistical wonks: The exit poll does only survey voters and not all Utah residents, but to argue that this causes an error, you have to show that a significant gender gap exists in voter turnout among Utah Mormons. There is no gender gap among all Utahns when you compare voters to nonvoters (see table 4b here). No turnout data exists for Utah Mormons by gender.
So, the 2010 exit poll suggests that 49.6%-52.0% of Utah Mormons are female. Meanwhile, the 2010 U.S. Census shows that 50.3% of voting-age Utahns are female. (To find that Census number for yourself, you need to start here; I can’t find a way to give a stable link.)
Incidentally, the exit poll estimated that 50.2% of voters overall in 2010 were female, which is amazingly close to the Census’s 50.3%. This is evidence that that ARIS survey may have oversampled females. Alternatively, it could be evidence the the exit poll oversampled males. There’s really no way to know which poll is correct, although the Census comparison does increase my confidence in the exit poll’s estimate. I suppose our only choice is to wait for additional, future polling work.
I thank Quin Monson for contributions to this post, and I thank the report’s authors for engaging me. Our discussion started in the comments thread on this post, then shifted to email. The author I was engaging with agreed with me to put relevant edits into this post and then delete the comment thread; by doing so, we hoped to minimize confusion for future readers.
This analysis was performed by Jordan Rogers, a student research fellow at BYU’s Center for the Study of Elections and Democracy, in collaboration with CSED faculty. The writing is mostly his. Inquiries about this research should come to me (Quin Monson).While Jordan cheers for BYU, his brother graduated from the University of Utah.
Since Brigham Young University (BYU) and the University of Utah (U of U) football teams played earlier than we’re all used to this year, researchers at BYU’s Center for the Study of Elections and Democracy decided to have a little fun with the rivalry. After all, there is a basketball game on Saturday.
In our September 2011 Utah Voter Poll (UVP), after asking a variety of political and demographic questions, we asked, “Who do you cheer for when the University of Utah plays Brigham Young University in athletic events?” (Note: we even alternated the order of the schools in the question stem to avoid biasing the results). Let’s take a look and see how BYU and U of U match up.
First of all, BYU and U of U supporters don’t have meaningful differences in educational attainment, income, or gender ratios. In a lot of ways, BYU and U of U fans are very similar.
But a stark difference arises when we look at politics. For example, respondents were asked to rate a variety of national and state politicians using a typical “thermometer” question. The table below shows average ratings for all respondents, each school’s fans, and a difference between the two. University of Utah fans have much more favorable opinions of Jim Matheson and Jon Huntsman, while BYU fans have more favorable opinions on the rest.
“Below is a list of politicians from the state of Utah. Please rate how favorably you feel toward each of the following officeholders on a scale from 0 to 100, where 0 is completely unfavorable, 100 is completely favorable, and 50 is neutral.”
Politician
Average Rating
BYU Average
U of U Average
Difference BYU – UofU
Gary Herbert
59
72
46
26
Orrin Hatch
51
63
37
26
Mike Lee
50
63
36
27
Rob Bishop
50
64
38
26
Jim Matheson
58
56
63
-7
“Below is a list of some of the candidates currently seeking the Republican presidential nomination. Please rate how favorably you feel toward each candidate on a scale from 0 to 100, where 0 is completely unfavorable, 100 is completely favorable, and 50 is neutral.“
Politician
Average Rating
BYU Rating
U of U Rating
Difference BYU – UofU
Mitt Romney
67
82
54
28
Jon Huntsman
57
53
63
-10
Michele Bachmann
31
40
21
19
Rick Perry
31
38
24
14
Ron Paul
40
41
36
5
However, before we make too much of these differences, it might be good to check whether the differences actually reflect the different partisan leanings among BYU fans and U of U fans. There are far more Republicans among BYU fans (85% GOP) than among U of U fans (40% GOP); however, even if we look only at self-identified Republicans, we still find that the two fan bases differ in their favorability ratings. The table below again compares the average ratings from each set of fans, but this time, only Republicans are included. Republican BYU fans have more favorable opinions of Gary Herbert, Orrin Hatch, Mike Lee, Jason Chaffetz, Rob Bishop and Mitt Romney, while Republican U of U fans have a more favorable opinion of Ron Paul. Jim Matheson received identical ratings from Republican BYU and U of U fans, perhaps explaining his uncanny ability to win reelection in Utah’s 2nd Congressional District.
Ratings Among Republicans Only
Politician
Average Rating
BYU Average
U of U Average
Difference BYU – UofU
Gary Herbert
71
76
63
13
Orrin Hatch
64
67
56
11
Mike Lee
67
68
65
3
Rob Bishop
65
68
60
8
Jim Matheson
54
54
54
0
Ratings Among Republicans Only
Politician
Average Rating
BYU Average
U of U Average
Difference BYU – UofU
Mitt Romney
81
86
77
9
Jon Huntsman
52
51
52
-1
Michele Bachmann
43
43
41
2
Rick Perry
41
40
41
-1
Ron Paul
42
39
43
-4
Overall, these data suggest some substantial political differences between U of U and BYU fans. While these differences get smaller among Republicans only, they don’t quite go away. In other words, there appear to be political differences among Republicans in Utah that can only be explained by knowing which school an individual cheers for. In that spirit, we reveal that we cheer for BYU and wish the BYU basketball team good luck on Saturday.
Our semester at BYU is winding down–classes ended on Thursday and final exams begin on Monday. As part of BYU’s Center for the Study of Elections and Democracy (CSED), we have organized the CSED Research Lab to provide a setting for CSED faculty and a few of our best students to interact about our research. This semester, each student has worked with a faculty member to produce a blog post for Utah Data Points. During the next week, we will post a few student-written contributions for what we hope will be a regular “student week” on Utah Data Points every semester.
Our first student post (to follow shortly) will help basketball fans get in the mood for Saturday’s BYU/Utah basketball game. Over the next week we’ll post about trends in Tea Party support and their importance for policy debates as well as some analysis of survey “feeling thermometers” that include Governor Gary Herbert and Representative Jim Matheson. We should note that a post earlier in October about whether legislators’ voting records match well with their constituents’ views was produced by Robert Richards, one of our CSED Undergraduate Research Fellows. For all of the posts during student week the analysis and writing we’ll post has been primarily done by the students, but we’ve ultimately checked and approved the work, so please direct inquiries (or complaints) about the research to us (the faculty members).
These students are doing very good work. Stay tuned…
As a result of recent redistricting, Representatives Fred Cox (R) and Janice Fisher (D) now find themselves in the same Utah House district and are likely to run against each other in November 2012. Let’s compare their legislative voting records.
I’ve written a few similar posts in recent weeks, such as my comparisons of Carl Wimmer and Stephen Sandstrom and of Ben McAdams and Ross Romero. Let’s give this series a name. From here out, I’ll put “cage match” in the title. If you’d like to suggest other legislators to compare, send me a note.
Rep. Janice Fisher has served several years now, but Rep. Fred Cox is a relative newcomer. The 2011 legislative session was his first. As such, this comparison draws on only 611 votes in which both Cox and Fisher participated.
The comparison
Cox and Fisher disagreed on 111 of 611 votes (that’s 18.2%). Given that these two belong to different parties, that may seem low. (It also puts into context my earlier findings that Republicans Wimmer and Sandstrom disagree only 10% of the time, and Democrats Romero and McAdams disagree only 4.5% of the time.)
Bear in mind, however, that most votes in the Utah legislature are decided by overwhelming majorities. It’s more informative to focus on the “close” votes. By “close,” I mean that fewer than 55 of 75 Representatives voted together on the winning side. Only 105 of the 611 total votes were close by this standard.
On close votes, Fred Cox and Janice Fisher disagree 57% of the time. (That’s 60 of the 105 close votes.) That should leave plenty for them to talk about during the election. Cox has voted “no” to Fisher’s “yes” 34 times; Fisher has voted “no” to Cox’s “yes” 77 times.
Bear in mind that I’m counting votes, not bills. If the House votes on different versions of the same bill more than once, then that counts as separate votes for purposes of this analysis. For example, Cox voted against HB 116 twice, and Fisher voted for it twice (note a correction to that sentence). Since bills often change between votes, I count each vote separately.
Examples of disagreements
I’ll list a few examples. First, I’ll list the four times where Cox or Fisher was the lone dissenting vote:
HB454, “State Hospital Revisions” (Cox was the only “no”)
HB22, “Fire Prevention and Fireworks Act Amendments” (Fisher was the only “no”)
SB237, “Pollution Control Facility Amendments” (Cox was the only “no”)
SB16S1, “State Tax Commission Tax, Fee, or Charge Administration and Collection Amendments” (Cox was the only “no”)
Now, I’ll list the five most divisive votes where Cox and Fisher disagreed. Each of these was decided by a margin of 6 or fewer votes in the 75-member chamber.
HB339, “Charter School Enrollment Amendments” (failed in a 37-37 vote; Cox “yes,” Fisher “no”)
HB491S2, “Alimony Modifications” (failed in a 35-38 vote; Cox “yes,” Fisher “no”)
HB89S1, “Protection of Children Riding in Motor Vehicles” (passed 39-35; Cox “no,” Fisher “yes”)
SB45, “Wireless Telephone Use Restriction for Minors in Vehicles” (failed 32-38; Cox “no,” Fisher “yes”)
Folks interested in easy-to-read summaries of current social science research should check out Journalist’s Resource. The site’s goal is to make life easier for journalists by pointing them toward interesting studies that can lead to engaging newspaper articles. I’ve had it in my feed reader for a while. If you follow Utah Data Points, odds are good you would like Journalist’s Resource.
An interesting example in their feed today: A new study by a sociologist finds that people who get actively involved in a church’s social activities (summer camps, youth programs, women’s organizations) end up believing more strongly in the church’s teachings. The study, written by a Baylor researcher, focuses on Protestants and Catholics.
We are professors of political science sharing academic research relevant to Utah. Posts are not peer reviewed and may discuss work in progress that is subject to future revision. Learn more. Each post reflects only its author’s views.
The 2010 exit poll suggests that 49.6%-52.0% of Utah Mormons are female. Meanwhile, the 2010 U.S. Census shows that 50.3% of voting-age Utahns are female.
After fruitful discussion with one of the authors of the report discussed here, I have made some changes to this post. I point out more significant updates with a brief “update” note, but I’ve made a few smaller changes without pointing them out.
I just read an article in the Tribune claiming that the gender gap is widening among Utah Mormons. The article is based on a recent report by two sociologists. It’s possible that the gap is widening, although another recent survey causes me to think twice.
For background, I’ll start by talking about what a margin of error is and how it should be interpreted. Then, I’ll present evidence from a separate survey that suggests a different conclusion.
What is a margin of error?
Every 10 years, the US government conducts a Census. They contact every person in the country and ask them several questions. Because they don’t use a sample, but attempt to talk to the entire population, they do not calculate a margin of error. They simply report the average age, income, gender, etc. of the entire American population.
But survey researchers don’t talk to the entire population. They talk to a sample of 500, 1000, or 2000 people. Because pollsters sample (unlike the Census), they cannot say exactly what a number is. They can only estimate. Using mathematical proofs, we know what the margin of error for any given sample will be. As your sample size grows, your margin of error shrinks.
Here’s the statistics from the report, as reported in the Tribune:
The Tribune article is quoting an analysis of the 2008 American Religious Identification Survey that was written by two sociologists. Here’s the relevant part of the report:
“Mormonism, like most Christian denominations in the United States, has a surplus of women. In 1990, this surplus was more pronounced among Mormons outside Utah, where 54.9% of Latter-day Saints were female, compared to 52.5% in Utah. By 2008, a dramatic shift had occurred. While the male to female ratio actually narrowed somewhat in most of the nation, it widened significantly in Utah. Females now outnumber males in Utah 3 to 2 [among Mormons].”
The report then goes on to try to explain the “dramatic shift.” The trouble is, we first need to be sure that the shift from 52.5% to 60.0% was large enough to get out of the margin of error.
(Update: This paragraph revised.) The report itself doesn’t give the margin of error for both numbers, so I did my best to estimate them. With a sample size of 270, the margin of error on the 2008 poll would approach 6%. (For fun, you can use this link to calculate a margin of error for any sample size). The report didn’t specify the sample size from 1990, but based on other numbers in the report, I estimated it at around 600.1 That would produce a margin of error of 4%. Thus, it may be that the 1990 survey estimated that 48.5-56.5% of Utah Mormons are female, while the 2008 survey estimated that 54%-66% of Utah Mormons are female. If the margins of error overlap like that, it’s possible that the true percentage has been 55% or so all along.
(Update: This paragraph added.) After corresponding with one of the report’s authors, I am less concerned about this problem. He shared with me some additional details of their analysis. In statistical jargon, it does appear that the difference is “statistically significant” between 1990 and 2008. That is, after taking account of the actual sample sizes and margins of error, it does appear that the authors are justified in saying that the gender gap in 2008 was larger than in 1990, at least based on the surveys they use.
After corresponding with the authors, I no longer question the authors’ interpretation of the 1990 and 2008 statistical data. Their interpretation was sound.
Evidence from another data source
Even in a perfectly orthodox statistical analysis, and even with a perfectly designed survey, we know from statistical theory that we’ll have an incorrect result in 1 out of every 20 tests. That’s just the nature of social science research. It’s true of my own research, and it’s true of this report. The trouble is, we have no way of knowing whether one particular analysis is one of those false positives.
The best way to detect a potential false positive is through replication. (Of course, if two surveys disagree, we still don’t know which one is the “1 in 20” error, so the more replication, the better.)
So let’s look at another survey and see what we find. The 2010 Utah Colleges Exit Poll, a statewide sample of voters, found that 50.8% of Utah Mormons are female. The exit poll reached 6,717 Utah Mormons, a very large sample, resulting in a small margin of error (1.2 percentage points). Thus, the exit poll estimates that the percentage of Utah Mormons that is female is somewhere between 49.6% and 52.0%. That sounds pretty close to the 52.5% reported back in 1990.
For statistical wonks: The exit poll does only survey voters and not all Utah residents, but to argue that this causes an error, you have to show that a significant gender gap exists in voter turnout among Utah Mormons. There is no gender gap among all Utahns when you compare voters to nonvoters (see table 4b here). No turnout data exists for Utah Mormons by gender.
So, the 2010 exit poll suggests that 49.6%-52.0% of Utah Mormons are female. Meanwhile, the 2010 U.S. Census shows that 50.3% of voting-age Utahns are female. (To find that Census number for yourself, you need to start here; I can’t find a way to give a stable link.)
Incidentally, the exit poll estimated that 50.2% of voters overall in 2010 were female, which is amazingly close to the Census’s 50.3%. This is evidence that that ARIS survey may have oversampled females. Alternatively, it could be evidence the the exit poll oversampled males. There’s really no way to know which poll is correct, although the Census comparison does increase my confidence in the exit poll’s estimate. I suppose our only choice is to wait for additional, future polling work.
I thank Quin Monson for contributions to this post, and I thank the report’s authors for engaging me. Our discussion started in the comments thread on this post, then shifted to email. The author I was engaging with agreed with me to put relevant edits into this post and then delete the comment thread; by doing so, we hoped to minimize confusion for future readers.