The Washington Post's Richard Morin Lies About His Poll Results

In a Tuesday 6/27 WaPo article subheaded, "Poll shows growth in support for Bush," it turns out that only three of the six poll results cited as evidence for that are statistically significant.

And digging deeper into the poll results, which contained a few dozen comparisons that could have potentially supported or contradicted that headline, I found only a half-dozen or so that supported the headline, and four that contradicted it, but the vast majority of the results were not statistically significant either way.

That means that, if the poll results changed from one month to another, the change was too small to be able to say that popular sentiment changed.

A few things you should know about standard errors (SE's) and margins of error (MOE's), before we start.

1) The MOE is just a constant multiple of the SE. Using a 95% confidence interval, which is what the WaPo seems to be using, the MOE = 1.96 * SE. Morin says his poll was based on a sample of 1,000, and has a 3% MOE; actually, the MOE is between 3% and 3.1% in most full-sample cases.

2) The SE of a difference between two numbers or percentages x and y is sqrt(SE²(x) + SE²(y)). If you don't like formulas, don't worry about it: if the two quantities have the same SE, what is means is that the SE of the difference is 1.414 times the SE of either quantity. That applies to the WaPo poll numbers. And the same multiple applies to the MOE. The MOE for a difference would be between 4.25% and 4.4%.

3) If you cut the sample size in half, the SE and MOE also go up by a multiple of 1.414, the square root of 2. So a difference between two half-sample results would have a MOE between 6% and 6.2%.

4) (2) above applies to statistically independent quantities. Some numbers move in parallel, and others move in opposite directions. For instance, when Bush's approval numbers go up, his disapproval numbers, big surprise, go down.

The SE of a difference between pairs of numbers like this - that move about the same amount, only in opposite directions - is twice the SE of either of the quantities, for obvious reasons: if Bush's approval changes by 5 points (say from 33% to 38%), then (approval - disapproval) changes by 10 points, in all likelihood, as his disapproval number goes from 65% to 60%, with the difference going from 32% down to 22%. It's the same thing, just expressed in a way that makes it look twice as big. Ditto the MOE, which would be between 12% and 12.4%.

Oh: Morin didn't tell you that, did he?

Sorry to have let the cat out of the bag.

Morin cites six trends to bolster the claim that Bush's support is growing, but only three are statistically significant, and you have to look at two of those just the right way in order to call them for Morin. Here's the breakdown:

1) Increase in overall Bush job approval, from 33% to 38%. Significant: MOE is 4.25%. (Poll Q.1)
2) Which party best able to handle Iraq: From 50-36 Dem to 47-41 Dem. Not significant: MOE is 12.1% on difference of spreads for half sample. (Q.6a)
3) Which party best able to handle terrorism: from 46-41 Dem to 39-46 the other way. Just barely not significant - same MOE as (2). However, the drop from 46% to 39% in Dem support is significant (6% half-sample MOE) so we'll give him the benefit of the doubt. Significant. (Q.6b)
4) Which party best able to handle the economy: From 52-34 Dem to 52-39 Dem. Not significant: Same MOE as (2). (Q.6d)
5) U.S. making significant progress toward civil order in Iraq: from 43-56 Y/N to 48-49 Y/N. Just barely not significant - same MOE as (2). However, the 7-point drop in the percentage of people who think Iraq isn't making significant progress IS significant, since the half-sample MOE on that is 6%. So, just like with (3), we'll give Morin the benefit of the doubt and call this significant. (Q.13)
6) Increase in approval of job Bush is doing in Iraq: from 32% to 37%. Not significant: 6% MOE, half sample. (Q.2a)

I emailed Morin on Tuesday inviting his comments, but since I haven't heard back yet, I'm posting. If my analysis is wrong, he can correct me after the fact, rather than before.

Morin should clearly state which margins of error apply to which comparisons he uses in his article. And it would be a good idea if, in the poll results themselves, he indicated which comparisons with previous months' results are significant, and which ones aren't. Casual readers are going to think the 3% margin of error applies to everything, and it doesn't. That's practically begging for people to draw the wrong conclusions from your polling.