Mellman: The folly in questions people can’t answer

I’ve been privileged to work with many campaign managers over the years, nearly all of whom have been wonderful. Most of them taught me important lessons.

At least once a cycle though, I have some variant of the following conversation:

Campaign Manager: I really need our poll to find out XYZ.

Me: I understand why finding out XYZ would be very important, but polls just cannot tell us that.

CM: But I really need to know. Just ask a question.

Me: I appreciate your need to know, but unfortunately your need to know is unrelated to voters’ ability to tell us. We’ll get a number, but it won’t mean anything.

CM: Just ask the question. I need to know.

Last week’s topic — likely voters — is one example of the substantive issues under discussion in these dialogues.

Where people get their information is another.

The payoff for knowing is huge. However, the possibility of finding out, accurately, from a poll is remote.

As with turnout, questions can be asked. Answers will be given and transformed into numbers. It’s seductive; it looks like science.

But we now know definitively that answers to questions about where people obtain their political information are so inaccurate as to be worthless.

A team of distinguished researchers led by David Rothschild of Microsoft and Tobias Konitzer of Predictwise analyzed the relationship between survey responses to media usage questions used by pollsters and hard behavioral data about what people actually do, gleaned from their electronic devices.

Just as voter files offer hard evidence about whether someone did or did not cast a ballot, we have data from television set-top boxes, computers and mobile devices that tell us, with some certainty, what media people tune into and for how long.

These hard data completely undermine any claim to accuracy made by poll questions on media usage.

For example, in February 2019, the usually useful Navigator poll found 34 percent of voters watched Fox News “a few times a month or more.”

But according to the data supplied by their electronics, just 18 percent of voters watched Fox News even once for six or more minutes during that month.

In short, Navigator overestimated the number of voters watching Fox by about 2 1/2 times.

It’s not just Navigator and it’s not just cable news.

A vast Knight Foundation survey queried 19,196 adults about how they “stay up on the news.”

Forty-one percent indicated they read news via social media links.

Again, the hard data during the same month the survey was fielded found just 9 percent of Americans consuming news that way, an overestimate of 450 percent.

Pew asked how often their respondents “get news from a social media site (such as Facebook, Twitter or Snapchat.)”

The hard data show poll responses exaggerate the facts by two to three times.

You don’t trust these researchers? You find flaws in their methods?

Try another set of studies, this time by Facebook researchers, together with colleagues from Michigan and Georgia Tech. With direct access to Facebook’s servers, they knew with some certainty how much time people were spending on their platform.

Comparing the server data to responses from ten different poll questions about Facebook usage revealed all the survey items produced results at wide variance with reality.

The “most accurate” question yielded answers far off the mark from 60 percent of respondents. Ninety percent, or more, of respondents were far off in their replies to half the questions.

Most respondents over-reported how much time they spent on Facebook, while under-reporting how many times they visited.

On open-ended questions, respondents overestimated their time on Facebook by nearly two hours a day!

You want a number from your pollster? Who is likely to vote? Who is spending how much time on which media? How many people will be persuaded by which message?

Pollsters can give you a number.

You want an accurate number?

That’s a different story.

When your pollster tells you there are things voters can’t accurately tell us, believe her or him.

Mellman is president of The Mellman Group and has helped elect 30 U.S. senators, 12 governors and dozens of House members. Mellman served as pollster to Senate Democratic leaders for over 20 years, as president of the American Association of Political Consultants, and is president of Democratic Majority for Israel.