Wednesday, July 11, 2012

Latest BC Poll, some issues I have

As many of you who read this know I have increasingly major issues with how polling is conducted in Canada and the lack of any real critical analysis of what is going on in the polls.

In BC this week there has been a new poll from Angus Reid and I have my concerns about it.

  1. I am not convinced that their online panel is representative of the voting public in BC - the pollster needs to show more of their work to give me the confidence in the sampling process.   Who answers online polls and what is their motivation for signing up in the first place?  Ideally they need to conduct some old style polls and online ones and see how close their model is.  For the moment I will assume the online method they use is no worse or better than by telephone.
  2. The margin of error reported is based on the total of 801 responses and not on the smaller samples such as decided voters.  They also do not report in the +- 0.5 percentage point margin of error due to the rounding of the results, this is relevant because they report their margin of error to the tenth of a point but not the actual results.   If this poll follows the norm of 80-85% of the respondents having an opinion, the actual margin of error is 3.75 to 3.87 percentage points.   Add in the rounding error and the actual margin of error on the headline results is more like 4.25 to 4.37 percentage points which is not only significantly higher than the reported 3.5 percentage points, it means a much flatter bell curve.
  3. The pollster does not indicate the size of the samples for the breakout groups, though from my calculations of it, it is very hard to see any breakout samples that are large enough to provide useful data.  As an example the Vancouver Island numbers should have been from only about 140 respondents which is a margin of error of 8.28 percentage points, add to that the rounding error and it becomes 8.78.   Providing very small samples with no indication of the margin of error is not responsible.
  4. They state the data is weighted to reflect the demographics of BC from the census.   This increases the margin of error but that is not reported.  I have no way to measure how much this changes the margin of error all I can say is that the more weighting there is the large the margin of error becomes.
  5. Weighting for census data also means the pollster is assuming voter turn out is not any different based on any demographics, this is a very flawed assumption.  The older you are the more likely you are to vote, the richer you are the more likely you are to vote, and finally the more educated you are the more likely you are to vote.  All indications are that older, more educated and wealthier voters are more likely to be on the right side of the spectrum.  Weighting for the census demographics is likely overstating NDP and Green support but I have no data to show how much of an impact it would have in this poll.
  6. The pollster asks about how people who voted NDP or Liberal in 2009 would vote now but do not give us any idea of how many of the 801 respondents this represents.  Based on a weighting that reflects actual voting in 2009, this would mean the largest sample was no more than 188 people.  Any measure of current vote intent based on their 2009 vote would have a very large margin of error, 7.2 percentage points for the 2009 Liberal voters and 7.5 for the NDP.  They do not ask who people who did not vote in 2009 are supporting.   This should represent a weighted sample of close to 400 people.
  7. The Angus Reid polling series is out of step with the only real data we have, the by-elections earlier this year.  In their March poll and the latest one the Liberals and Conservatives are nearly tied, in May the Liberals had a small lead.   If this were true the BC Conservatives should have won in Chilliwack Hope and not come third.   The by-elections point more to the BC Conservatives being in the mid teens in support and the Liberals nearer to 30%.   The pollster does not address why their numbers are at odds with actual results.
  8. The attention to detail does not seem to be there in the poll.    According to what I have been told, the pollster asked not about the BC Conservatives, but the BC Progressive Conservatives.   Asking the wrong name must have some impact on the result but what I do not know.

Apparently a screen capture from the July Angus Reid poll
Post a Comment