Pollsters confront tough survey landscape after 2020 flubs

Academic and nonpartisan pollsters surveying voters ahead of key election contests this year are trying new technology and massaging their existing methodology in hopes of correcting the errors that led them to underestimate Donald Trump’s level of support in the 2020 election.

But those pollsters on the front lines today are contending with some of the most difficult-to-survey contests in recent memory, from a California recall with dozens of candidates on the ballot to a mayoral election in Buffalo where the incumbent is running as a write-in candidate.

The tweaks and corrections pollsters are making today will inform the broader survey research industry, from the academic centers that partner with national and local media outlets to the partisan pollsters who make millions from their work with Democratic or Republican groups and candidates as the midterm elections approach.

“You’ve got to make sure that you’re not just simply correcting the mistakes of last time, which are idiosyncratic to last time,” said Patrick Murray, director of the Monmouth University Polling Institute. “But obviously there are errors that you’ve got to take a close look at after 2020.”

Most pollsters agree about the fundamental problem, laid out in a detailed study by the American Association for Public Opinion Research of 2,858 surveys conducted during last year’s presidential contest. That study found the widest error in any election cycle going back 40 years: State-level polls overstated President Biden’s advantage over Trump by more than 4 percentage points, and they understated Trump’s overall level of support by about 3 percentage points.

Many pollsters even agree on the cause of that problem: Those surveys chronically underrepresented voters with lower levels of education, who were likely to back Trump, or who had never voted before.

The AAPOR survey did not reach a definitive conclusion about why polls were off by such a margin.

But it landed like a bomb among a beleaguered industry that is struggling to rebuild trust with the public after the 2016 elections, in which most polls correctly showed Hillary Clinton winning the popular vote but missed Trump’s late surge in states like Wisconsin, Michigan and Pennsylvania, and the 2020 elections, in which the ultimate results were similarly correct though the margins were off.

“We definitely took the problems that the AAPOR report detailed quite seriously,” said Scott Keeter, a senior survey advisor at the Pew Research Center. “It seems clear that there is some differential nonresponse going on. That is that either supporters of Trump were less likely than average to be available or to cooperate with polls, or that the Biden supporters were just exceptionally willing to do so.”

Pollsters in the field today are trying everything they can to answer the fundamental question that all political polling seeks to answer: Who, exactly, is going to show up to vote? And just as media has become so fragmented that outlets must now reach readers or viewers through varied channels, so too are pollsters trying to figure out how to reach an electorate that is increasingly unwilling to answer a call from an unrecognized number.

“We all deal with the repercussion of the trust in polling, and that’s what we have to rebuild,” said Spencer Kimball, director of Emerson College Polling. “That’s why we’re looking at new technologies to improve the composition to get it more representative.”

Some pollsters are experimenting with text messages with a link that deliver respondents to a web-based survey, though those surveys tend to underrepresent older voters who are less text-native than younger generations. Some are experimenting with web-based polling, though those surveys may be too self-selecting to create a random sample.

If there is any methodology that is falling increasingly out of favor, it is the one that created the modern survey research industry: Live operators calling people on landlines just after they get home from work.

“Using traditional live operators to call people at 5:30 is not how people are communicating today,” Kimball said.

At the same time, some pollsters believe the single biggest factor that led to the misses last year has effectively resolved itself: The voters they were most likely to miss last year were ardent fans of Trump. Trump’s absence from the ballot this year — and in next year’s midterm elections — makes those voters that pollsters have found hardest to reach unlikely to rejoin the electorate any time soon.

“We are not clear yet whether those people are only Trump voters,” Murray said. “But that seems to be the most likely hypothesis. They are only Trump voters. They voted in 2016 or they voted in 2020 specifically because Donald Trump’s name was on the ballot.”

Trump’s absence from the ballot this year is not making pollsters’ lives any easier, because they are such difficult races to poll.

In California, testing whether voters want to recall Gov. Gavin Newsom (D) in an election next month may be straightforward, but testing which candidate voters prefer to replace him is a herculean task: There are 46 candidates on the ballot, and there is no way to ask voters about each one of them.

Kimball, who has surveyed the race, said many voters in his polls opt to write in Newsom’s name — a choice that would invalidate their vote if they do so on their actual ballot, because Newsom cannot replace himself.

Kimball has also tested the race to be Buffalo’s next mayor. Mayor Byron Brown lost the Democratic primary to activist India Walton, but Brown is running as a write-in candidate, so his name will not appear on the ballot.

Perhaps the clearest indication of the success pollsters will have in addressing the flaws from 2020 will come in Virginia, where voters will elect a new governor this year.

Three polls conducted this month all agree on one thing: That former Gov. Terry McAuliffe (D) leads Republican nominee Glenn Youngkin. Where they diverge is by what margin. A Christopher Newport University poll out this week showed McAuliffe taking 50 percent of the vote, 9 points ahead of Youngkin. A Roanoke College poll showed McAuliffe up eight points, but at just 46 percent. A poll conducted by Virginia Commonwealth University showed a much closer race, with McAuliffe leading by three points, well within the poll’s margin of error.

Pollsters say they are taking heart in the results of one question that is particularly salient: The share of people who report they are vaccinated against the coronavirus. Many polls show the share of Americans who say they have been jabbed aligning closely with data released daily by the Centers for Disease Control and Prevention.

“The alignment of the vaccination numbers in the polls with the actual numbers from the CDC is a very encouraging point,” Pew’s Keeter said.

In spite of the errors of 2020, and the struggle to rebuild trust among a polarized electorate in which people have markedly less contact with those of opposite political persuasions, pollsters are in no way packing up shop — the demand for their work remains.

“Our appetite for public opinion is as strong now as it’s ever been. Despite the misses of 1948, despite the misses of 2016 or 2020,” Kimball said. “Survey research works. There’s a margin of error, there’s a range, but you can tell within four or five points whether a position is popular or not.”

“The problem we have as an industry,” he said, “is we present it as an exact score.”