Before voting began on Election Day, nearly every major poll was predicting a Hillary Clinton win by 2-4 percentage points. When the smoke cleared Wednesday morning, Donald Trump had won.
In the wake of Trump’s surprise win, arguably the biggest fascination has been the failure of the polls. Politico asked, “How did everyone get it so wrong?” Fusion asked how it went “so, so, so wrong?” Harvard Business Review wrote that pollsters were “completely and utterly wrong.”
Yes, the polling was wrong—but the reasons why are numerous, and nuanced, and will take a long time to fully parse and understand. In addition, it wasn’t just the polls that went wrong, but also the media’s interpretation of the polls.
1. Polls did not fully account for the Shy Trump Voter
One of the biggest theories as to what the polls missed was the idea of “shy Trump voters” who didn’t want to say when polled that they were planning to vote for Trump, but always knew.
White women, in particular, proved to be a surprise: 53% of them voted for Trump overall, led by those without a college degree, who went for Trump by a 2-1 margin. White women with a college degree went for Clinton, but only barely, by six percentage points. “There’s your shy Trump vote,” tweeted Kristen Soltis Anderson, a pollster at Echelon Insights.
Anderson later added that a bigger problem than secret Trump voters was “a phony mirage of a Clinton vote.” Trump got fewer votes than McCain did in 2008 and Romney did in 2012 and won anyway, because too many Democrats didn’t vote.
Indeed, polling also fails to account for turnout, which was the lowest overall it has been since 2000. (Latino turnout was up from 2012 and skewed toward Clinton, but not by enough to beat Trump.) All non-white ethnic groups went for Clinton, as did millennials—but not enough of them voted.
As Harvard Business Review points out, “People tend to say they’re going to vote even when they won’t… the failure of a complex likely voter model is why Gallup got out of the election forecasting business.”
2. Polling methods need to change
As much as big data (and the technology to sift through it) has advanced, our methods of gathering data are still dated. Most of the national polls are still done by landline telephone. And that has been a problem for over a decade now.
In 2003, Gallup wrote a post about the falling response rates in polls. If you start with a target sample size of 1,000 households, Gallup wrote, at least 200 households fall out because they are businesses or non-working numbers. Of the 800 left, another 200 “may be unreachable in the time frame allocated by the researcher… household members at these numbers may use caller ID or other screening devices and refuse to answer.” Now you’re down to 600, of which 200 more people may pick up the phone but refuse to participate in the poll. Suddenly, the sample size has shrunk from 1,000 to a mere 400 households. Declining to pick up the phone, or declining to participate in the poll, may have been a particular problem with this election polling.
The shrinking sample size is a significant problem. As pollster Anderson tweeted, the “only way you can bring down margin of error is to raise sample size.” That’s not easily done.
In an interview with Bloomberg, Iowa pollster J. Ann Selzer pointed to “the continuing barrier of the lack of landlines, the erosion of landlines” as a particular problem this cycle. Bloomberg wrote it in October: “Your mobile phone is killing the polling industry.” And Matthew Nisbet at The Breakthrough noted back in 2012, “Other under-reported sources of error also factor into a poll’s accuracy, including the greater reliance on cell phones.”
Online polling is a newer method, but has its own problems. Trump campaign manager Kellyanne Conway said back in August, after a Trump dip in the polls, that the candidate “performs consistently better in online polling where a human being is not talking to another human being about what he or she may do in the elections.” The Washington Post pointed out that this wasn’t the case overall—on average, Trump wasn’t doing better in online polls than in telephone polls.
However, a Morning Consult post from Nov. 3 (with nearly the now-suspect headline, “Yes, there are shy Trump voters. No, they won’t swing the election”) pointed out that Trump was doing 1% better in online polls than phone polls, a difference small enough to be dismissed. But here was the key line in the Morning Consult post: “Trump’s edge over Clinton online instead of in phone polling is especially pronounced among people with a college degree or people who make more than $50,000… more-educated voters were notably less likely to say they were supporting Trump during a phone poll than in an online survey.” That was the exact slice of voters that went for Trump more than anyone expected.
So it isn’t black-and-white whether phone or online polls are better, and it isn’t clear that phone polls should die; but it is clear that methods of polling need to evolve and improve, and that the best route to get as many data sets as possible is a combination of different methods.
3. The bigger failure was interpretation of the polls
After an initial immediate backlash to the polls, a newer narrative is already emerging: the polls didn’t fail as terribly as everyone is saying they did.
Many are pointing out that Clinton looks likely to win the popular vote (although barely, and by a smaller margin than Gore won it in 2000). If Clinton does win the popular vote by around one percentage point, then polls that showed Clinton winning by two or three points were only one or two points inflated. Moreover, polls come with a margin of error that in many cases did cover the eventual difference.
The problem is that in a 140-character media landscape, margin of error is often left out, or squeezed into posts and articles as an asterisk.
The election polls were actually off by less than Brexit polls were off. And Nate Silver of FiveThirtyEight pointed out on Thursday morning that this year’s polls were in fact more accurate than in 2012. That year, polls generally predicted a slim Obama win margin of 1 percentage point, and he won by 4 points. This time, the polls gave Clinton a margin of 3-4 points, and she looks likely to win the popular vote by 1 or 2.
Of course, that defense won’t exactly quell outrage over the polling (just look at the replies to Silver’s tweet), because the polls in 2012 didn’t call the wrong winner. There’s a big difference between Obama winning by a larger margin than polls said he’d win by, and Trump winning when polls said Clinton would win.
And to be sure, a fair retort to Silver and others claiming that the polls weren’t that wrong is that the result here was binary: polls could either predict the right winner or the wrong winner. Almost all of them predicted the wrong winner.
Polls are estimates. They are a projection of what appears likely to happen, within a margin of error. But we take them too literally. As Fairleigh Dickinson University professor Peter Woolley told Bloomberg, “We tend to over-report the accuracy of the poll, and tend to forget very quickly that it’s an estimate within a range.” The biggest problem with the polls this time around, then, wasn’t actually the polls, but our interpretation of them.
Because the vast majority of the polls (all of them but two, from USC/LA Times and IBD/TIPP) had Clinton winning, the media and the public counted on a Clinton win, ignoring the fact that most polls had her winning only slightly, and many had a margin of error that allowed for the opposite result. The volume and noise drowned out nuance.
In a September article in The Atlantic (appropriately headlined, “Taking Trump seriously”), Salena Zito wrote of Trump, “The press takes him literally, but not seriously; his supporters take him seriously, but not literally.” The media spent time picking over everything Trump said as though he were serious, when he often wasn’t, and didn’t take him seriously as a legitimate threat to Clinton; his voters didn’t worry too much about each individual shocking sound bite, but took him seriously as a candidate.
In a column published after Trump’s victory, Maureen Dowd of The New York Times pointed to Zito’s line as a “prescient” one, and it truly was—it describes not just the result of the election, but the problem with how the media embraced the polls. Pundits – and the public – took the polls literally.
Many are now asking whether polls are even useful if they can be so wrong. Does the Trump surprise win kill the polling industry? Hardly. Polling isn’t going anywhere, but the methods need to improve, and we must temper our embrace of the predictions they yield. They are only that: predictions.
—
Daniel Roberts is a writer at Yahoo Finance, covering sports business and technology. Follow him on Twitter at @readDanwrite.
