Lost in a Sea of Numbers
I am a huge fan of Talking Points Memo and, in particular, TPM Election Central, so the following must be taken in the spirit of admiration and bonhomie in which it is offered:
You guys don’t know how to report on polls. Seriously. You’re utterly clueless.
All it takes is a shiny new press release to prompt the TPM guys to slap up a 48-point headline announcing the latest devastating news from the front lines of the electoral process.
The problem is that these polls do not share a common methodology and are often of uncertain provenance, so they aren’t measuring the same thing (and often, I suspect, aren’t even measuring the things they intend to), yet they are paraded across the screen to the viewer as though they represent succeeding chapters of a coherent narrative.
In the recently completed PA Primary, for example, TPM reported on poll results from no fewer than 78 polls over the course of the contest.
6 of these polls were one-time wonders. Although sometimes by a recognizable media outlet, a one time poll is as useless as a Greenpeace bumper sticker on Dick Cheney’s limo. Without a couple of attempts, you can’t even tell if the polling methodology is stable. These were:
EMILY’s List (Ed. Note: Seriously, WTF?)
Mason-Dixon
Suffolk
Temple
TIME
Times/Bloomberg
The following chart summarizes the remainder. Comparing to the final results of the contest (54.5% HRC, 45.4% BHO), the metrics you want to check are A) who got the spread about right, as close as possible to a 9.1 point spread (note to TPM: 9.1 is not 10, you can look that up), and you also want to see B) who had the smallest margin of ‘Undecideds’ … we are trying to project how people will vote, not how long they can dither. The gray semicircle on the y-axis represents the Actual finish (centers on the 9.1 point spread … Undecideds are Zero on election day).
The winners are Quinnipac, SurveyUSA, and Zogby/Newsmax. The lesson here is that whoever is covering the News Desk should consider not publishing anybody that hasn’t yet got at least a little bit of history, like this, of doing well.
The losers show a variety of problems. By 4/20, PPP had somehow determined that the race was going to Obama. In a dead heat that is an excusable mistake, but HRC won by over 9 points! Similarly, ARG’s bubble seems to be filled with helium. Muhlenberg was actually on its way to nailing it, but their last poll was on 4/2. The intern must have gone on spring break.
Mark Penn has demonstrated that numbers are more often than not a favorite weapon of flim-flam men and charlatans. Which is a shame, because basic statistics can be a great revealer of truth, but only if shared by a fourth estate that knows how to select and report on a quality product.
1 Comment »
RSS feed for comments on this post. TrackBack URI
Leave a comment
Line and paragraph breaks automatic, e-mail address never displayed, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>
Very interesting to see it graphed out. Seriously, you should start an official Poll rating service - like Morningstar but for Polls!
Comment by mike — April 30, 2008 @ 9:36 am