Should We Ever Trust Polls Again? 

The credibility of polling took a massive hit this month, as polling averages were off the mark by double-digits in Michigan’s Democratic Senate primary and Wisconsin’s Democratic gubernatorial primary.  

Before these contests, in a July column on the heels of the Graham Platner implosion, I flagged my concerns with the state of the polling industry: “since high-quality polling has become more expensive, as it has become harder to get potential respondents to pick up the phone, fewer media outlets in an age of declining local news are producing their own surveys.” That has created a vacuum being filled by new pollsters without proven records and the campaigns themselves which try to shape perceptions by publishing so-called internal polls. 

And as we’ve seen, polling can impact reality. Drying up fundraising. Dissuading volunteers. Sapping staff morale. Prompting candidates to quit.  

When Mandela Barnes suspended his campaign for Wisconsin governor on July 30, he said, “It’s become very clear who our nominee is going to be,” implying Assemblywoman Francesca Hong was a lock. The last five polls published before he quit—all taken after Lieutenant Governor Sara Rodriguez dropped out over a campaign finance debacle and Milwaukee County Executive David Crowley reversed his own withdrawal—averaged Hong at 40 percent, Barnes at 21, and Crowley at 12. Crowley would end up beating Hong in the primary, 39.8 to 39.3.  

When Maine Governor Janet Mills suspended her campaign for U.S. Senate in late April, bowing to what looked like a Platner juggernaut, she acknowledged, “I very simply do not have the one thing political campaigns unfortunately require today: the financial resources.” But a sitting governor would not have any problem raising money if she wasn’t behind by an average of 33 points in the prior month of polling, most of which came from sources tied to the Platner campaign. One wag on X observed, as the surprising Wisconsin results rolled in, “So was Janet Mills even actually down when she dropped out?” Probably, but maybe not by an insurmountable 33 points, and maybe there was time to mount a comeback. 

Does this mean that polling is beyond redemption? No, we have primaries this cycle where the poll aggregates accurately depicted the state of the race, such as Georgia’s Republican Senate primary and California’s open gubernatorial primary. There is even an internal pollster, Public Policy Polling, which, as noted in Wednesday’s Washington Monthly newsletter, has produced several on or close to the mark surveys this primary season. (But nobody’s perfect: in the California gubernatorial, Public Policy Polling was a little too bullish for its client Tom Steyer and bearish on the primary winner Xavier Becerra.)  

But we all seem to have forgotten that polls are imperfect snapshots of public opinion and are not designed to be predictive, especially weeks before Election Day when many voters are still undecided or only have soft support for a particular candidate. For example, the final set of polls in the Texas Democratic Senate primary, held on March 3, all showed a victory for State Representative James Talarico. Yet most polls in January and early February had U.S. Representative Jasmine Crockett ahead.  

The Wisconsin gubernatorial primary was unusually volatile in the final month, with Crowley dropping out to endorse Rodriguez, then Rodriguez dropping out, then Crowley getting back in with the incumbent governor’s endorsement, then Barnes dropping out. Meanwhile, Hong sat for a series of awkward interviews focused on old social media posts. Voters could barely keep up with the constantly changing developments, so how could a pollster? Whatever failings fall on the shoulders of Wisconsin pollsters, poll consumers who foolishly treated the numbers as fixed and impervious to late movement also deserve blame. 

Ask any New York Knicks fan: Don’t look at the scoreboard in the third quarter and assume the game is over.  

Of course, in a basketball game, everyone knows the scoreboard is accurate. With polls, we don’t. Pollsters with long records of accuracy are harder to find and even the best polls still have margins of error. Furthermore, as we just saw in Michigan, there’s always the possibility of systemic polling error, either because pollster groupthink causes everyone to make similarly inaccurate assumptions about turnout demographics (“herding”) or because certain chunks of the electorate are more or less likely to participate in survey (“response bias”). 

After Donald Trump pulled off the upset of the century in 2016, the polling industry debated whether they missed “shy Trump voters” who didn’t want to admit their preferences or whether Trump simply won over undecided voters too late for polls to detect. We’ll never know for sure, but with Michigan primary polls seemingly overloaded with ideologically passionate college-educated Abdul El-Sayed voters and lacking in low-key working-class Haley Stevens voters, perhaps Trump voters aren’t the only hard-to-reach cohort.  

Modern technology may also be making it harder for pollsters to connect with older voters. Two decades ago, Barack Obama’s presidential campaign argued primary polls showing him behind Hillary Clinton were too reliant on landline phone users and were missing young voters who only use mobile phones. Now we face the opposite pitfall. Most households only use mobile phones which make it easier to screen callers and make it more expensive for pollsters to reach a sufficient number of respondents. Some pollsters are saving money with text-to-web methods, sending a text message with a link to a web survey, but that runs the risk of disproportionately excluding seniors.  

The odds of colossal poll misses like we saw in Michigan and Wisconsin are higher in primaries. Turnout mixes can vary widely from primary election to primary election depending on the type of candidates on the ballot and whether or not the race is seen as hotly contested with an uncertain outcome, making life very difficult for pollsters as they weigh their data. But we have had infamous general election poll whiffs in recent memory, such as the 2020 Senate election in Maine and 2016 presidential election in Wisconsin, requiring us all to remain on guard and avoid treating any data as infallible. 

That goes double for political prediction markets which are being dangerously legitimized, most glaringly by Kashi’s partnership with CNN. Kalshi rival Polymarket also recently forged a partnership with a new media company called Eventual. But these markets are not providing high-quality insights worthy of news coverage. Traders are largely reacting to shaky poll data, which is why on Monday Hong was given a 95 percent probability of winning in the Kalshi market. Similarly, as I noted in my last column, Kalshi suddenly gave State Senator Jarrett Keohokalole a 70 percent chance of winning the Hawaii First Congressional District Democratic primary after his campaign published an internal poll with him in the lead, just before he lost by 21 points. 

Not every poll is bunk. And parties still need poll data to help make informed decisions about where to invest resources in general elections. For example, this week an internal poll conducted by Public Policy Polling showed Kansas’ new Democratic nominee Senate Adam Hamilton—a prominent Methodist minister—just one point behind in the incumbent Republican Roger Marshall. Is that enough information for Democratic Party organs to shovel money into Kansas? As noted above, Public Policy Polling has built a strong track record this cycle, but no pollster is perfect and it has an interest in making its client look competitive. Democratic Party operatives will likely want to see additional polling as well as focus group or other qualitative data before making any decisions.  

For the rest of us, we should treat poll data with extra caution and not make any assumptions about who is going to win until the votes are counted. Celebrating too early can be painful. Ask the San Antonio Spurs.  

The post Should We Ever Trust Polls Again?  appeared first on Washington Monthly.