Think you know how to read a political poll? Think again.
It’s polling season again. News media is rife with stories about the results of polls trying to show who’s ahead in political races across the country. This morning I opened my local news feed to find this headline:
Evan McMullin, Sen. Mike Lee locked in nasty battle for Senate. What does latest poll show?
As a sociologist and retired statistics professor, I open these kinds of stories with immense caution. While the numbers reported tend to accurately represent the results, the headlines and descriptions of what the numbers actually mean often leave out important information at best, or at worst, misconstrue the results to imply that one candidate has an edge over another when in reality it’s impossible to tell.
This morning, I clicked on the headline to see what the polls said. Here’s the opening caption:
Utahns like McMullin a little more than Lee, but the independent challenger has a lot of voters who don’t seem to know him and he still slightly trails the Republican incumbent in the heated U.S. Senate race. (Stuart Johnson and Jeffrey D., Allred, Deseret News)
According to this, Evan McMullin “slightly trails” Mike Lee in the Senate race. Is this true? Let’s look at the numbers.
The numbers show that 41% of people polled said they’d vote for Mike Lee if the election were held today, and only 37% said they’d vote for McMullin. The next largest category is “Don’t Know” at 12%. At first glance it certainly looks like the opening caption is accurate. After all, 41% is greater than 37%, so can’t we just report that and move on?
Not exactly. Polls are based on random samples of people in a target population. In this case, the target population was registered Utah voters. As of this morning, just over 1.9 million people in Utah are registered to vote, but according to the note at the bottom of the graphic, the poll only reflects the opinions of 801 registered Utah voters. Pollsters can’t possibly survey every person in the target population, but thanks to statistics they don’t have to. While sampling is a valid scientific method of inquiry, results based on samples must be interpreted in their full context. This is what news organizations often neglect in favor of the eye-catching headline and the opening hook.
Let’s assume that Dan Jones & Associates (the firm listed as the source of the poll) knows what they’re doing and drew a statistically valid and reliable random sample. This means that the 801 Utahns surveyed are likely to represent the population of registered Utah voters with some chance of making a mistake. Statisticians use this chance of making a mistake to calculate a “margin of error.”
The margin of error gives a range within which we can be reasonably certain that the true percentage of the population lies on a given issue. Because the results are based on a single sample that’s one of an infinite number of samples that could have been drawn, we must interpret the results within the context of potential error.
To understand this better, imagine the polling firm put the names of all 1.9 million registered Utah voters into a hat. Then, they randomly pulled 801 names from that hat. That’s one sample. But they could just as easily have pulled a different set of 801 names from the hat. That would be another sample. If they did this 100 times (always putting all names back in the hat), there would be 100 different samples. The statistics reported in political polls show the results from just one of these potential samples. So, while one sample is a useful starting point for understanding public opinion, the margin of error uses statistical procedures to estimate the range of likely results with more confidence than a single reference point.
Let’s see how this works in practice. The poll indicates a margin of error of +/- 3.46 percentage points (see the note at the bottom of the graphic above). This means that with 95% certainty (more on that in a minute), the true percentage of registered Utah voters who support any of the given candidates lies within a range of 3.46 percentage points above and below the reported result. In other words, we can be 95% certain that if we subtract and add the margin of error to any given result we’ll get the range within which the true percentage is likely to lie. (This particular poll, as with most public opinion polls, uses a 95% confidence level, which means that the range calculated using the margin of error still has a 5% chance of being incorrect.)
While 41% of people in this particular sample said they’d vote for Mike Lee, the real percentage is likely anywhere from 37.54% to 44.46%. Similarly, 37% said they’d vote for Evan McMullin, but the real percentage is anywhere between 33.54% and 40.46%.
I’m a visual person, so here’s a graph showing the range of possible results. The horizontal bars show the percentage in the single random sample of 801 people who said they’d vote for each candidate. The black bars at the ends show the range of percentages that are equally likely to be the real percentages for the population.
You’ll notice that the two black lines overlap. This means that both candidates are equally likely to be ahead (or behind). It’s worth stating again that ANY percentage within the ranges of the black bars is equally likely. This means that all of the follow scenarios are equally likely:
Mike Lee: 37.54%
Evan McMullin: 40.46%
McMullin would win.
Mike Lee: 40%
Evan McMullin: 34%
Lee would win.
Mike Lee: 38%
Evan McMullin: 38%
Lee and McMullin tie.
So yes, 41% is greater than 37%, but 40.46% is also greater than 37.54% and 38% is equal to 38%. In short, we can’t say which of the two candidates would win if the election were held when the poll was taken.
Back to the opening caption of the news story, we see that the statement that McMullin “slightly trails” Mike Lee is incorrect because both candidates are equally likely to win or tie.
In short, if you want to understand public opinion polling, you need to subtract and add the margin of error to any results you see. If the ranges overlap, then you know that both outcomes are equally likely, as is a tie.
The headline that spurred me to write this article wasn’t as egregious as some. With a quick search, I found this headline on The Hill:
Lee leading McMullin by 4 points in Utah Senate race: poll
This kind of misreading of poll results misleads the public. I hope that by reading this article, you’ll do the math to understand what polls are actually telling us before simply accepting the headlines and news stories as truth.
