Polling the right people matters

Polling methodology has been a hot topic of late. But in all the
discussion about “random deletion,” weighting by vote, analytics and
dozens of other important issues, a basic principle has been
overlooked, a principle that could well be a key culprit in some recent
massive failures — you’ve got to poll the right people. And the right
people are the electorate.

You’d be surprised how often the wrong people get polled.

{mosads}Last year, The Forum of Fargo-Moorhead polled only likely primary voters, inaccurately concluding that Rick Berg (R) led now-Sen. Heidi Heitkamp (D) by 7 percentage points in the Senate general election. Surveying the wrong people yielded the wrong result. We polled general election voters for Heitkamp and never had her anywhere near that far behind (our final result was within a point of the actual vote).

More subtle, but no less problematic, is another form of this error that derives from the “likely voter” fetish. I don’t care about likely voters. I care about the likely electorate, and so should you.

Failure to take that advice could be at the root of inaccurate prognostications in the New York City comptrollers’ race, the Rochester mayoral race and parliamentary elections in British Columbia, Canada, this year. In two of these cases, pollsters focused on “likely voters,” missing the likely electorate by a mile. In the third, the pollster showed no concern about voters or electorates.

Let’s start with the New York City comptroller’s race between Scott Stringer (disclosure — our client) and Eliot Spitzer. A New York Times/Siena poll had Spitzer ahead by 15 percentage points less than two weeks before the election, while a slightly earlier NBC 4/Wall Street Journal/Marist poll gave the former governor an 18-point lead.

It sounds plausible until you realize that our polling had Stringer ahead the entire time and our final track was just about half a point off the final margin.

Why the difference? We focused on the likely electorate, Siena and Marist on likely voters. Our likely electorate was about 25 percent of registered Democrats. The actual turnout was about 23 percent. Siena was looking for likely voters using a carefully constructed model based on responses to 5 questions.

However, their likely voters constituted 67 percent of registered Democrats, three times the number that actually turned out. Marist’s likely voters were 59 percent of Democrats, also far more than voted.

Siena and Marist were talking to a lot of people who said they were registered Democrats (but may not have been) and claimed they were likely to vote (but didn’t). Those folks are quite likely to be different from the 23 percent who actually voted. They are less politically engaged and less likely to have known Stringer.

We saw evidence of the impact in our polling. The most frequent voters in our sample favored Stringer by 10 points, while less frequent voters were nearly tied.

It’s likely that true non-voters tended to favor Spitzer, and they made up the majority of Siena and Marist’s so-called “likely voters.”

Siena had the same problem in Rochester mayor’s race, where their final poll had the incumbent leading by 36 points, though he lost by 17 points days later. What was the actual turnout? Twenty-two percent. Siena’s likely voters implied a turnout of about 75 percent.

In British Columbia, Angus Reid didn’t bother with either likely voters or a likely electorate, and so its forecast of a 9-point victory for left-of-center New Democrats proved embarrassing when the party lost by 4 points. The actual turnout was 58 percent, but Reid seems to have assumed a 100 percent turnout, surveying “adults.”

Polls can get the right result without worrying about likely electorates when those who don’t vote would cast their ballots the same way as those who do.

But in low-turnout situations, there can be significant differences between voters and non-voters, and polling the right people matters.

Mellman is president of The Mellman Group and has worked for Democratic candidates and causes since 1982. Current clients include the Majority Leader of the Senate and the Democratic Whip in the House.