Why Polls Screw Up
And we are here as on a darkling plain,
Swept with confused alarms of struggle and flight
Where ignorant armies clash by night
- Matthew Arnold (”Dover Beach”)
Frantic analysts are scrambling today to rationalize the complete failure of the pollsters to call the Clinton victory in the NH primary. Here is the actual rationalization. They all got it wrong because they were all measuring the wrong thing.
This sounds trite, but it is a mistake made constantly. Statistical models are based on three very basic assumptions. A) That you have a clear understanding of how a mechanism worked for the period of time you are studying, B) That you have clean data measuring the major components of that mechanism, and C) That this phenomenon will continue to work the same way in the future.
The fact that there are dozens of polls, each with different results for the same phenomenon, demonstrates that there is often disagreement about A.
Everybody’s got a theory, and these range from careful deduction to wild-eyed wishful thinking – and sometimes a dash of outright fraud attempting to create a bandwagon effect in the me-too media or to sow the seeds of panic in the enemy camp. All of these models attempt to represent a different mechanism. For this reason alone, at least the better political blogs need to stop running minute-by-minute headlines of the every latest poll’s findings. It lends to the idea that they are all comparable, and they aren’t.
B, collection of clean data, is the bugaboo of statistics. Hard to do well, and often assigned as a menial task to luckless souls without the capacity, means, or context to do it well. Sets of non-homogeneous garbage data are routinely gathered up and analyzed, and the most sophisticated statistical tests to find a trend or patterned distribution among them are about as effective as taking an aspirin for a case of bubonic plague. Garbage in, Garbage out.
C is the most insidious of all. That the mechanism will work the same in the future is the idiot assumption behind almost every financial, economic, and political projection you will ever see. Idiot, because bundled into the reason for even DOING the projection is that, if you don’t like where the trend is heading (and in a political contest, wherever it’s heading is bad news for somebody), you can DO SOMETHING DIFFERENT. The forecast carries with it the seeds of its own undoing.
Team Hillary saw that terrible gap and they responded. I don’t always agree with the woman, but she and her staff are no dummies, and when Loss reared its head, they busily worked to change the game. Their opponents seemed eager to add their own ill-considered actions into the mix, the media was trumpeting each and every statement over the airwaves with the authority of Jove and the experienced news judgment of a pigeon … all of which creates a volatile mix in which surprising, and certainly unmodeled, things can happen.
I don’t argue that forecasts are meaningless, but they are not reality. They are mathematical tools normally far removed from their strictly defined mathematical aquariums, attempting to model phenomena usually not easily quantified. They are invaluable tools within the context of the Scientific Method, but outside a laboratory, they are ALL suspect, and we should treat them with the skepticism they deserve.
2 Comments »
RSS feed for comments on this post. TrackBack URI
Leave a comment
Line and paragraph breaks automatic, e-mail address never displayed, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>
To elucidate – or perhaps, to reiterate – your last paragraph: statistical analyses are models; elections are the real thing (unless, of course, the Republican party is counting the votes, and then all bets are off). The beauty of statistics is that in the hands of the right (or left) people, they can mean anything you want them to mean. I had a calculus teacher in high school named Doc Wagner. He could draw a perfect circle on the blackboard using his elbow as the center point and the rest of his forearm as the radius. He was a rotten calculus teacher, but he was a nice guy, He once told me, and I quote, “Figures don’t lie, but liars figure.” Every time I look at the results of a poll, I think of old Doc Wagner, and I envy him his perspicacity.
Comment by Steve Alber — January 10, 2008 @ 6:10 pm
Wasn’t it Doc Wagner who calculated the statistic that the average American has one ball and one tit?
Comment by Al — January 11, 2008 @ 9:33 pm