Mark Mellman: Poll bias takes many forms

Even after watching the previews, I am not sure how I will feel about the forthcoming Russell Crowe movie presenting a richly embellished account of the Noah story (turning 46 brief biblical verses into 132 minutes of Hollywood spectacular certainly requires a lot of embellishing). But I am sure that a “poll” covered in Monday’s Hollywood newspaper of record, Variety, will tell me nothing about how the film will be received. 

How a reputable media outlet was conned into even reporting the “poll” is a major mystery. First, it seems the poll wasn’t even a poll, but responses an organization received to a question placed on its website. Yet Variety’s headline screamed “98% of Faith-Driven Consumer Audience Dissatisfied with ‘Noah,’ Hollywood Religious Pics.” 

{mosads}The lead encapsulates the story, “Despite its biblical inspiration, Paramount’s upcoming ‘Noah’ may face some rough seas with religious audiences, according to a new survey by Faith Driven Consumers. The religious organization found in a survey that 98% of its supporters were not ‘satisfied’ with Hollywood’s take on religious stories such as ‘Noah,’ which focuses on Biblical figure Noah.”

How this audience, forgetting for a moment the unscientific nature of the sample, could have become dissatisfied with the film before it even opened and before respondents saw it is only an enigma if you don’t read the question on which all the hubbub is based: “As a Faith Driven Consumer, are you satisfied with a Biblically themed movie — designed to appeal to you — which replaces the Bible’s core message with one created by Hollywood?” 

A second-grader at Vacation Bible School can see the bias. The question is telling you that the Bible’s message is being replaced with Hollywood drivel and then asking you, as a person of faith, how you feel about it. 

Poll warriors, constantly on the prowl for biased questions, rarely find examples this egregious. However, in their haste to deconstruct polls, they sometimes miss vastly more subtle, but still significant, sources of bias. 

Witness the current survey-based lament over the level of scientific literacy in America. As United Press International reported, “According to a new survey by the National Science Foundation, nearly half of all Americans say astrology … is either ‘very scientific’ or ‘sort of scientific.’ ” To make us feel worse, that was contrasted with the 92 percent of Chinese who “think horoscopes are a bunch of baloney.”

In fact, the question tells us about something about Americans’ literacy, not about their scientific literacy. For questions to mean what we, who write them, think they mean, respondents have to understand them the same way we do. It turns out that lots of folks confuse astrology (unscientific) with astronomy (scientific). 

A study by Richard Landers, of Old Dominion University, found that 24 percent define astrology incorrectly, mostly as astronomy. Among those who defined it correctly, just 13 percent thought it was even “sort of scientific.” Of course, the question asked of the Chinese public is different and therefore not really comparable, but we seem to be in the same range when it comes to belief in the occult. 

And on a host of other questions measuring scientific knowledge, which went largely unreported, Americans outscore the Chinese by large margins. For example, 84 percent of Americans knew “the center of the earth is very hot,” compared to 56 percent of Chinese, while 83 percent of Americans know about continental drift, compared to 50 percent of Chinese.

Whether blatant or subtle, bias can creep into questions in a wide variety of ways. Remain vigilant, lest you be misled by voters’ misunderstanding of what is being asked of them.

Mellman is president of The Mellman Group and has worked for Democratic candidates and causes since 1982. Current clients include the majority leader of the Senate and the Democratic whip in the House.