Promoted by Steven D
At the start let me say what this isn’t. This is not an unskew the polls diary. In fact, this is the opposite: this is a data driven diary that is based on one simple table. These are the averages for all polls in October and November where the results found a margin of less than two points. Honestly I didn’t think I would find anything. But the evidence is clear. There is a consistent bias against Democrats.
Why am I the only one seeing this? Honestly, most don’t have data that goes back as far as I do (I think 538 goes back to ’04).
So why would this happen? In June of 2012 I wrote a diary entitled “Are cell phones resulting in biased polls against Democrats?”. In it I noted that minorities and the young were far more likely not to have a landline than the rest of the population. This became apparent in the aftermath of 2012.
Something else no one seems to have noticed. Every telephone poll of the national 2012 Presidential Election Understated Obama’s margin. The last time this happened was 1980! This is not new, though. The Generic Ballot polling was pretty bad in 2010 as well.
Look at the table. The average polling error for the Senate has been:
2012: -5.32
2010 -3.21
2008 -2.28
2006 -2.68
2004 2.33
2002 4.25
2000 -3.91
1998 -6.27
So in the last 4 elections, Democratic support has been underestimated by over 2 points!! In the last 8 elections 6 have seen significant understatement of Democratic Support in close races.
More tomorrow.
The results below are taken from my database of 11471 polls from 1998 to 2012.
I love it when you talk data to me.
speaking of which, anyone else watch/ listen to the MT Amanda Curtis/ Daines debate yesterday? she’s impressive.
I hope she has a GOTV operation that dominates Western Montana and Billings.
what are you thinking?
Is that the area that most needs GOTV?
Very useful — this should be promoted heavily. Thanks for compiling this
From a statistical standpoint, all of those errors could be completely random given the sample sizes of the polling involved on each poll.
Even given the fact that there is a run of errors all in the same direction.
Picking winner over loser is a coin-toss model. One would have to demonstrate significant departure from a run of 7 or 8 coin tosses to argue those results anything but random.
My hunch is that aggregating all of the polls requires paying attention to variance, skew, and kurtosis in order to make sense of whether there is systematic bias going on. Long ago I used to know how to do that estimation of bias. Maybe some folks still doing statistics can clear up whether this is just a random phantom observation or not.
Not that I wouldn’t like to see pollsters nailed dead to rights on biased polling. Because some do seem to add more secret sauce than is necessarily warranted.
My position is that polling does not accurately predict turnout. And that the systematic error in its assumed turnout patterns is hidden by random offsetting errors in the voting outcome, that is covered in reported MOE. That is, the polls are not and cannot ever be accurate on turnout because people’s intentions at the moment are affected randomly. Workload prevent people committed to voting from actually getting out. Child gets sick on election day. And on and on.
but not over a number of them. I will run the numbers to find out how many were in one direction – it is possible the mean is distorted by outliers – but I don’t agree with you.
Take 2012 national polling. EVERY poll was under the margin. There were 9 polls in the final RCP average. The probability of that result if there was no bias in either direction is .0019, or about 2 tenths of a percent.
When the data is reduced to a binary error-nonerror string, independently arrived at data can easily have runs of 50 and 60 in one direction. That is a low probability situation, but not so low as to be excluded without further examination of the data.
The problem with scoring polls is that there really is a small sample of them. You are looking at the level of the poll itself, not at the underlying population of respondents. The question is whether the pollster’s error was random or the result of bias.
Not if the results are random. Probablity theory is pretty simple – if the results are random then even a run of 10 in a row is extremely unlikely.
There are two types of errors in polling:
The second type of error results when you take a poll at time Y, and an event causes people to change their minds and vote differently on election day.
It is this second type of error – which is much more common than people think occurs – that I will write about today/tomorrow.
I’m concerned that we’re trying to unskew the polls…
I’m with Tarheel in refusing to remember how to calculate the bias, but that’s the kind of pattern that usually comes out significant, related not to purposeful skewing but uncontrolled factors like the cell phone thing. Especially remembering that the exception years 2002 and 2004 are the “9/11 changed everything” years when the whole electorate was suffering from PTSD-related psychosis.
It’s my understanding that pollsters have eliminated the possible landline/cell phone bias.
Telephone polling was good enough when almost everybody had a landline and answered their phones. It was also relatively cheap to do polling. Then came answering machines, followed by caller ID, and then the cell phones. It’s surprising that there isn’t more sampling bias in the polls considering the population most likely to respond to polling.
Should add that while I’ve only been paying close attention to polling since 2002 (2000 at the POTUS level), the good enough polls, with a soupcon of secret sauce in the very close elections, accurately predicted the results. Didn’t add my two cents on leftie blogs before the 2002 election because my analysis didn’t conform to the consensus that DEMs would do well. So, I thought that I had something to contribute. (A lot of statistics in two disciplines.) Learned quickly enough that it wasn’t welcome. Understandable in 2004 when I could see the looming train wreck; so, I bit my tongue and took it off-line. But when 2006 looked great, 2008 good, and 2012 good enough, there seemed to be a preference for fretting.
The 2014 election may be the most difficult to project since 2000. As it doesn’t seem as if it’s going to break in favor of Democrats or Republicans, or incumbents or challengers.