Polls, lies, damned lies and statistics

Throughout the world of political reporters and pundits, the blogosphere and social media — left, right and center — the conventional wisdom was impressive: Obama’s negative ads on Bain Capital and outsourcing were working in the 12 battleground states where they were exclusively running. 

First, two weeks ago, came the Wall Street Journal/NBC poll, which triggered an avalanche of reporting reinforcing the themes that the Obama negative ads were hurting Romney and helping Obama big time. The national results were a statistical dead heat (47 percent Obama, 44 percent Romney). But in the 12 battlegrounds — Colorado, Florida, Iowa, Michigan, Nevada, New Hampshire, New Mexico, North Carolina, Ohio, Pennsylvania, Virginia and Wisconsin — where Obama has spent tens of millions on negative ads since April, Obama was ahead 50 percent to 42.

{mosads}The Journal/NBC poll also showed that in the battleground states, Romney suffered a net increase of 11 percent in negative attitudes toward his business experience — from late May (36 percent favorable to 36 percent unfavorable) to late June (30 percent favorable, minus 6 percent, to 41 percent unfavorable, plus 5 percent). The cause reported by the Journal and NBC: the Obama negative ads.

But not so fast.

Voters were asked in the national survey whether they would feel more or less positive toward a businessman who managed a company that specialized in restructuring and selling companies; i.e., to anyone watching Obama’s TV ads — which were seen by most of the country on either TV or the Internet — those words could only describe Bain Capital. The answer between January 2012, before the negative ads ran, and June, after they ran, was no change at all (from minus 6 percent in January to minus 5 percent in June).

In addition, I could find no disclosure in the reporting of the Journal/NBC poll of the subsample total number of voters surveyed in the 12 battleground states. The total surveyed nationally was just 819 registered voters — with a fairly large plus-minus margin of error of 3.4 percentage points. It is therefore possible that the number of voters surveyed in the subsample among 12 battlegrounds was a fraction of 819 — meaning a considerably higher margin of error, which could make the 50-42 percent “lead” for Obama in the battlegrounds statistically meaningless.

USA Today’s Susan Page reported that among the 1,200 voters surveyed in the 12 battlegrounds, 8 percent of that total, or 96 voters surveyed, said they “changed” their minds because of the ads, from one candidate to the other. Of those who changed, 76 percent (of the 8 percent, or 73 voters across 12 states) said they switched to Obama, and 16 percent (15 voters across 12 states) said they switched to Romney. Page described this data as evidence that Obama’s negative ads were the “clear” winner. 

That’s not so clear to me.

First, the subsample is very small — 8 percent of the total sample of 1,200, or 96 people spread out over 12 states — even though the 5-to-1 change ratio favoring Obama is statistically significant. 

Second, to conclude the ads “caused” the change, it is required to have a control group, i.e., the same number of voters surveyed who have not seen the ads. For example, if the results among that control group show the same or a similar percentage of changes favoring Obama, that would negate the inference that the ads “caused” the change. Without a control group, you simply cannot be sure of the cause.

Third, in April, a USA Today/Gallup poll in battleground states showed Obama ahead 51 percent to 42. But in this late-June sample, the results were Obama, 47 percent, to Romney at 45 percent. In other words, after Obama spent an estimated $68 million or more over Romney in the battlegrounds on negative ads during this time period, Obama lost a net 7 points. 

As Republican pundit Rich Mullens noted in his “must-read” daily blog, Mullings:

“According to the RealClearPolitics average of polling in 12 swing states, Obama has leads outside the margin of error in only Pennsylvania, New Hampshire and Nevada, Romney leads in two (North Carolina and Missouri) and the rest are essentially tied, such as Obama’s 0.8 percent lead in Florida.”

Doesn’t USA Today/Gallup’s own comparison between April and June pool results in the battlegrounds as well as this RealClearPolitics data demonstrate that, if anything, Obama’s negative ads have actually hurt … Obama?

The moral of the story: When you see all the media jumping in one direction in interpreting polls, beware. 

When they don’t tell you the total sample, or subsample in the battlegrounds, or the margin of error of the subsample, or whether there were control groups to determine whether the negative ads “caused” the change of minds, beware. 

In other words, when you read poll results interpreted by a political media prone to rush to judgment all at once in the same direction, remember what Mark Twain frequently said, quoting former British Prime Minister Benjamin Disraeli’s famous expression:

“There are lies, damned lies, and statistics!”

Davis, the principal in the Washington law firm of Lanny J. Davis & Associates, which also specializes in legal crisis management, served as President Clinton’s special counsel from 1996-98 and as a member of President George W. Bush’s Privacy and Civil Liberties Oversight Board.  He is the author of the book Scandal: How “Gotcha” Politics Is Destroying America.