Mellman: The problem with interpreting poll crosstabs

Last month, Gallup generated a great deal of attention for a startling finding: During the month of October, President Joe Biden’s approval rating apparently tumbled 11 points among Democrats — his core constituency. 

Presumably based on conventional wisdom, but no actual evidence, Gallup attributed that drop to opponents of the president’s support for Israel in the wake of Hamas’s murderous Oct. 7 attack. 

However, although it elicited no attention, come to find that Gallup’s polling in November recorded an 8-point jump in the president’s approval among Democrats, nearly reversing the October fall. 

So much for Gallup’s initial explanation. It’s hard to argue that Biden’s support for Israel hurt him in the immediate aftermath of the attack but is now helping him. 

Moreover, it’s clear the October measurement was a statistical outlier. From January through September this year, the president sported an average 84 percent approval rating from fellow Democrats. In October, the reading was 75 percent, and then in November it was back within a point of the average, at 83 percent. 

The “drop” to 75 percent was most likely noise, whether because of a small subsample or just an odd one. It is unlikely to have reflected reality. 

The commentary, however, reflects a broader problem of interpretation. Pollsters and their antagonists regularly dive into subgroup data from individual surveys to make broad pronouncements about races. As Gallup learned, it can be dangerous.  

It’s hard enough to get the topline numbers right these days. Subgroup numbers (“crosstabs” in the vernacular) suffer from all the same problems and more.  

Poll aggregators like 538.com and RealClearPolitics have tried to solve the topline problem by averaging a large number of polls to get better horserace estimates, which also prove more stable, rendering them less interesting to those fascinated by wild swings. 

Former pollster Adam Carlson has done a great service by applying simple aggregation techniques to subgroups — averaging the crosstabs from November polls, then comparing them to the average of final 2020 readouts from the exit polls, Catalist and other post-election data providers.  

For a year or more I’ve been arguing that the 2024 presidential contest will be tight, and Carlson’s arithmetic suggests movement away from President Biden compared to 2020. But the story these data tell is a bit different from those normally trotted out.  

According to the November polls, the biggest movement away from Biden is among Black voters, with whom he enjoyed an 83-point margin in 2020. It now seems to have shrunk to 52 points — a stunning 31-point drop.  

Is it real? 

Well, it’s not just weird or small samples.  

However, another blogger, Dan Guild, demonstrated that “pre-election surveys have consistently underestimated the Democratic margin among African Americans. In fact, every poll that released crosstabs in 2016, 2020, and 2022 underestimated the Democratic margin.” 

Guild calculated that error at 6 points in 2016, 17 points in 2020 and 27 points in 2022.   

None of this is to suggest the president’s campaign doesn’t need to pay greater attention to African Americans — even if today’s margin proves to be 27 points smaller than on election day, Biden would still be doing less well with the Black community than he did in 2020 imperiling his chances in key Electoral College states.  

But it should lead us to read these data with some caution. 

The second biggest decline in Biden’s margin is not among self-identified liberals or Democrats, but rather among political independents. 

Taken together, November’s polling pegs Biden’s margin among independents 27 points lower than it was on election day 2020. Victory also requires a stronger showing with this critical segment.  

Third on the list is Latino voters, among whom Biden’s margin shrunk by nearly 20 points. 

Our research, reported here earlier, indicated that the Latinos most at risk were those with the lowest levels of ethnic identification. So ethnic-focused appeals are less likely to be effective with this critical community.  

In short, be careful interpreting crosstabs, and even more important, don’t plan a campaign strategy around applying conventional wisdom to the crosstabs in any one poll. 

Mellman is president of The Mellman Group and has helped elect 30 U.S. senators, 12 governors and dozens of House members. Mellman served as pollster to Senate Democratic leaders for over 20 years, as president of the American Association of Political Consultants, a member of the Association’s Hall of Fame, and is president of Democratic Majority for Israel.