The 2010 exit poll suggests that 49.6%-52.0% of Utah Mormons are female. Meanwhile, the 2010 U.S. Census shows that 50.3% of voting-age Utahns are female.
After fruitful discussion with one of the authors of the report discussed here, I have made some changes to this post. I point out more significant updates with a brief “update” note, but I’ve made a few smaller changes without pointing them out.
I just read an article in the Tribune claiming that the gender gap is widening among Utah Mormons. The article is based on a recent report by two sociologists. It’s possible that the gap is widening, although another recent survey causes me to think twice.
For background, I’ll start by talking about what a margin of error is and how it should be interpreted. Then, I’ll present evidence from a separate survey that suggests a different conclusion.
What is a margin of error?
Every 10 years, the US government conducts a Census. They contact every person in the country and ask them several questions. Because they don’t use a sample, but attempt to talk to the entire population, they do not calculate a margin of error. They simply report the average age, income, gender, etc. of the entire American population.
But survey researchers don’t talk to the entire population. They talk to a sample of 500, 1000, or 2000 people. Because pollsters sample (unlike the Census), they cannot say exactly what a number is. They can only estimate. Using mathematical proofs, we know what the margin of error for any given sample will be. As your sample size grows, your margin of error shrinks.
Here’s the statistics from the report, as reported in the Tribune:
- In 1990, a survey found that 52.5% of Utah Mormons were female.
- In 2008, a survey of 270 Utah Mormons found that 60% of Utah Mormons are female.
“Mormonism, like most Christian denominations in the United States, has a surplus of women. In 1990, this surplus was more pronounced among Mormons outside Utah, where 54.9% of Latter-day Saints were female, compared to 52.5% in Utah. By 2008, a dramatic shift had occurred. While the male to female ratio actually narrowed somewhat in most of the nation, it widened significantly in Utah. Females now outnumber males in Utah 3 to 2 [among Mormons].”
The report then goes on to try to explain the “dramatic shift.” The trouble is, we first need to be sure that the shift from 52.5% to 60.0% was large enough to get out of the margin of error.
(Update: This paragraph revised.) The report itself doesn’t give the margin of error for both numbers, so I did my best to estimate them. With a sample size of 270, the margin of error on the 2008 poll would approach 6%. (For fun, you can use this link to calculate a margin of error for any sample size). The report didn’t specify the sample size from 1990, but based on other numbers in the report, I estimated it at around 600.1 That would produce a margin of error of 4%. Thus, it may be that the 1990 survey estimated that 48.5-56.5% of Utah Mormons are female, while the 2008 survey estimated that 54%-66% of Utah Mormons are female. If the margins of error overlap like that, it’s possible that the true percentage has been 55% or so all along.
(Update: This paragraph added.) After corresponding with one of the report’s authors, I am less concerned about this problem. He shared with me some additional details of their analysis. In statistical jargon, it does appear that the difference is “statistically significant” between 1990 and 2008. That is, after taking account of the actual sample sizes and margins of error, it does appear that the authors are justified in saying that the gender gap in 2008 was larger than in 1990, at least based on the surveys they use.
After corresponding with the authors, I no longer question the authors’ interpretation of the 1990 and 2008 statistical data. Their interpretation was sound.
Evidence from another data source
Even in a perfectly orthodox statistical analysis, and even with a perfectly designed survey, we know from statistical theory that we’ll have an incorrect result in 1 out of every 20 tests. That’s just the nature of social science research. It’s true of my own research, and it’s true of this report. The trouble is, we have no way of knowing whether one particular analysis is one of those false positives.
The best way to detect a potential false positive is through replication. (Of course, if two surveys disagree, we still don’t know which one is the “1 in 20″ error, so the more replication, the better.)
So let’s look at another survey and see what we find. The 2010 Utah Colleges Exit Poll, a statewide sample of voters, found that 50.8% of Utah Mormons are female. The exit poll reached 6,717 Utah Mormons, a very large sample, resulting in a small margin of error (1.2 percentage points). Thus, the exit poll estimates that the percentage of Utah Mormons that is female is somewhere between 49.6% and 52.0%. That sounds pretty close to the 52.5% reported back in 1990.
For statistical wonks: The exit poll does only survey voters and not all Utah residents, but to argue that this causes an error, you have to show that a significant gender gap exists in voter turnout among Utah Mormons. There is no gender gap among all Utahns when you compare voters to nonvoters (see table 4b here). No turnout data exists for Utah Mormons by gender.
So, the 2010 exit poll suggests that 49.6%-52.0% of Utah Mormons are female. Meanwhile, the 2010 U.S. Census shows that 50.3% of voting-age Utahns are female. (To find that Census number for yourself, you need to start here; I can’t find a way to give a stable link.)
Incidentally, the exit poll estimated that 50.2% of voters overall in 2010 were female, which is amazingly close to the Census’s 50.3%. This is evidence that that ARIS survey may have oversampled females. Alternatively, it could be evidence the the exit poll oversampled males. There’s really no way to know which poll is correct, although the Census comparison does increase my confidence in the exit poll’s estimate. I suppose our only choice is to wait for additional, future polling work.
I thank Quin Monson for contributions to this post, and I thank the report’s authors for engaging me. Our discussion started in the comments thread on this post, then shifted to email. The author I was engaging with agreed with me to put relevant edits into this post and then delete the comment thread; by doing so, we hoped to minimize confusion for future readers.