Wednesday, July 18, 2012

How surveys can be so very wrong

I was made aware of this table last night when engaging with someone on twitter about who votes in elections.   As soon as I saw the table I could see that it was highly flawed.  It also highlights the fact that a large portion of the public lies on polls, a large enough portion to make the results of most polls less accurate than dead reckoning.

This is from the 2008 General Social Survey - Selected Tables on Social Engagement.   I am highlighting this one table to show how fundamentally wrong the research is.    This is a table of the voter turn out in BC based on what people answering the survey answered they did in the last election.   The numbers are wildly far off of reality.

The provincial election this surveyed asked about was the 2005 one, problem is that only 1,762,450 voted in that election and not 2,420,000 as the survey suggests.  

The last federal election would be the 2006 one and the turnout in BC was 1,827,183 and not 2,459,000 as the survey suggests.

Since we know this survey is so dramatically off on this one data point we have hard data on, we can not rely on any of the rest of data as reflecting what the public thinks.    The spending on this exercise by Statistics Canada was a complete waste of money.  They should explain why they are so wrong and if they can not explain it they need to stop doing any work like this.

While it is a waste of resources to have done this survey at all, the bigger problem is that people rely on the data in the survey as having some connection to reality.  The whole dataset needs to be removed from the internet and replaced with a statement from Statistics Canada along the lines of the work was fundamentally flawed and gives no quantitative information of relevance.

So why would the voter turnout numbers be so much higher in the survey than in reality?   I suspect that it is because many people who did not vote planned on voting and do not want to admit they did not vote.   They lied on the survey, a much more common occurrence than most people are willing to admit happens in polling.

Pollsters need to figure out how to account for people that lie in polls.  While I do not have the rigorous data to show the details of a bias, the best I can get to is that public in many polls is biased against the government and for the main opposition.

Post a Comment