Friday, April 27, 2012

A polling failure in Alberta?


The polling in the Alberta election was miles off of the actual result and I think makes for a very good election to study how well pollsters do and why they are more often wrong than right.  I would love to see all the pollsters make their raw data available to academic researchers to see if there are reasons for why the pollsters were so far off.

In the Alberta election the polling numbers very consistent in the last ten days.  If we take the last ten polls and drop the best and worst result for each party we get:
PCs 32-36%, Wildrose 40-42.8%, Liberals 9.6-12%, NDP 9.7-13%

Actual results
  PC    Lib    WRP   NDP   AP   Other
43.95% 9.89% 34.29% 9.82% 1.33% 0.72%

Only the NDP and Liberal ranges capture the actual election results, and then at the outside edge

Median results over last 11 polls
PC 33%, Wildrose 41%,  NDP 11%,  Liberals 10%

Looking first at Forum Research since the released a poll the day before the election, had the largest sample sizes and conducted six polls
  Date    PC    Lib   WRP   NDP    AP   Other Poll size

March 26  31    12    41    11     2     3     1069  
April 3   29    13    43    10     3     3     1689
April 9   31    10    43    11     2     2     1781
April 16  33    10    40    12     2     2     1639
April 21  32    10    41    13     2     2     1950
April 22  36    10    38    12     3     1     1949
+- elect  +7.95 -0.11 -3.71 -2.18 -2.67 -0.28
I look at their numbers and feel like the final poll was done to be closer to the results?  But even their final poll, which was out of sync with everything else, they only come to within the margin of error of one party, the Liberals.

You may argue that the Alberta Party results are higher than they got because they did not run a full slate, but the pollster should have accounted for that and only allowed AP as an answer in ridings where they were running.

There is also no strategic voting going in the last days of the election because the numbers for the NDP and Liberals stay consistent through the election and in the case of the Liberals are the only numbers they got right.

So where did the PC vote come from and when did it happen?   Actually the better question is where did the Wildrose vote go?  The final Forum poll, which showed the PC rising and Wildrose falling on the weekend before the election, showed that the biggest shift was of 2008 PC voters being more likely to vote PC than Wildrose in 2012 than they were in the poll the day before.

The possiblities:

  1. A lot of people planning on voting Wildrose decided to vote PC at the last minute.
  2. A lot of people said they would be voting Wildrose but never really planned on voting for them but did not want to admit this to anyone.   No one has ever studied how many people lie to pollsters and who is more likely to lie and who not.   
  3. The sample of people responding to polls is not accurate reflection of who actually voted - we can reasonably certain this is a factor because a much higher percentage of the public answered with a decided opinion than actually voted.
  4. Many Wildrose voters simply decided to stay home.    This one can be tested to some extend by looking at vote turnout in various ridings in this election and the last election.  Should I have the time I will do this.

Next, looking at Campaign Research
Date       PC   Lib   WRP   NDP  Other  Poll size

March 26  30.3 13.0  39.6  11.6  5.5    924
April 3   28.4 11.3  45.5  10.2  4.6    840
April 11  34.4  9.6  42.8   9.7  3.5    880
April 19  34 11    41    11    3      ????

They did not poll over the weekend, but they had enough time to capture right wing missteps of the Wildrose candidates.  Their April 11th poll comes very close for the Liberals and the NDP, but in their next poll those two parties gain.

Abacus Data
Date         PC  Lib  WRP  NDP Other Poll size

March 26–28  28   16   41   12   3    1036
April 2–4    31   12   43   11   3     876
April 9–11   29   10   46   12   2     900
April 18–19  31   12   41   13   3    1076
Their final poll was conducted four days before the election and they were well outside of the range for all the parties.

Abacus, like Campaign Research and Forum all conducted polls after the wing nut incident by the Wildrose candidate on April 16th.  They should have captured the mood shift if there was one.

Abacus did have some interesting data on how people voted in 2008 and how they intended to vote in 2012.  Among 2008 Liberal voters, 29% of them intended to vote for the PCs, 11% for Wildrose, and 13% planned on voting NDP.


Now Think HQ Public Affairs
Date         PC Lib  WRP  NDP  AP  Other Poll size

March 22–25  36  13   33   13   2   3      1320
April 2–3    30  11   43   12   3   1      1050
April 9–10   29  12   43   13   2   1      1223
April 17–18  33  11   41   11   3   1      1425

Once again, much the same story of the other pollsters when it comes to the numbers.


Now looking at Leger
Date        PC   Lib   WRP   NDP   AP  Green Other Poll size

March 22–25 37    12    34    11    2    0    6     1215
April 2–4   33.9   9.5  41.3  11.7  2.2  0    1.4    986
April 5–8   34.2  12.5  35.5  13.2  2.7  1.3  0.6    902
April 13–16 36     9    42    10    2    0.3  1     1200


Angus Reid, Ipsos Reid and Return on Insight each conducted one poll during the election.  Their results were broadly in line with all the others.


Either we have to accept there was a 150,000 vote shift in the last three days of the election or that there are systemic problems to do with polling.

No comments: