Gallup overhauls polls after 2012 failure

Gallup on Tuesday unveiled a sweeping review and overhaul of its polling methodologies after producing data that was skewed toward Republican presidential nominee Mitt Romney in the 2012 election.

The pollster’s final estimate of likely voters last year gave Romney a 49 percent to 48 percent edge over President Obama. That was vastly off the mark, as Obama won the popular vote by 3.85 percentage points.

{mosads}Frank Newport, Gallup’s editor-in-chief, said the “skewed distribution” in favor of Romney “was something going on in the entire industry,” but he acknowledged, “Gallup was at the bottom of that.”

Gallup’s final estimate for the election was 1 percentage point more in favor of Romney than the industry average.

Newport presented Gallup’s findings to a group of reporters and political scientists at the firm’s Washington headquarters. He detailed areas where the polling outlet went wrong in 2012 and said Gallup is seeking input from industry professionals on ways it could improve its methodology.

Gallup has partnered with the University of Michigan for the top-to-bottom review of its operations. In a written post-mortem, the pollster said it’s not averse to making “major revisions or even a replacement model” if needed to produce more accurate data.

The primary area of focus will be on Gallup’s estimations of likely voters. Gallup has one of the most closely watched daily poll averages, and its filter of likely voters is supposed to bring an added degree of accuracy over the registered voter model.

But Newport said declining response rates, increases in early voting, targeted campaign efforts to reach new voters, and the penetration of new technologies have dated aspects of Gallup’s likely voter model.

Newport was flanked by political scientists from three major universities, as well as Gallup’s own methodologists and research analysts.

The panel of experts frequently pointed to the Obama campaign’s sophisticated operation as an example of how polling has failed to keep pace.

Newport called the Obama campaign “an interesting window” into new areas of statistical exploration. He said, just like the Republican National Committee, polling outlets are beginning to study the methods it used for outreach and data collection.

Among the changes Gallup is considering is reweighting its data to fully account for the outsized influence a handful of swing states now have in presidential elections.

Newport said the Obama campaign treated battleground states “as their own universe,” and suggested polling of them might need to be given more weight than states that are traditionally a lock to go either red or blue.

In addition, Newport said the logistics of Gallup’s deadline-oriented daily polling caused the firm to consistently under represent some portions of the country.

“Simulations indicate that the underrepresentation of specific time zones within regions led … to Gallup’s underestimation of Obama’s vote share,” the report concluded, citing the Pacific time zone in particular.

Gallup plans to expand its race and ethnicity categories by allowing voters to choose from five races or ethnicities. The pollster will begin weighting by four of those categories, rather than just black and Hispanic, as it did in 2012.

The firm will also calibrate its cellphone-to-landline ratios by moving from a “listed landline” sample, which it said produced “older and more Republican respondents,” to a “random digital dialed” sample, in hopes of smoothing out the technology divide.

“None of these factors are large in and of themselves,” Newport said. “But they are significant enough that we think they made a significant difference in our overall assessment in who was going to win the presidential election last fall.”

In conducting the review, Gallup was able to identify some aspects of its polling that didn’t contribute to the 2012 variance. For instance, the Gallup brand produced “no house effect,” according to the report, and in fact, the review found voters were more likely to respond to a survey conducted by the well-known operation.

2024 Election Coverage

• 2024 Election News
• 2024 Election Polling Data
• 2024 Presidential Election Forecast

Gallup also found that its decision to leave specific third-party candidates off its survey didn’t skew the results, and that its randomization of candidate name order successfully eliminated potential biases in survey responses.

“When the next presidential election rolls around, we think we’ll certainly be in a position at the accurate end of the spectrum,” Newport said.

— This story was updated at 8 p.m.