In BC this week there has been a new poll from Angus Reid and I have my concerns about it.
- I am not convinced that their online panel is representative of the voting public in BC - the pollster needs to show more of their work to give me the confidence in the sampling process. Who answers online polls and what is their motivation for signing up in the first place? Ideally they need to conduct some old style polls and online ones and see how close their model is. For the moment I will assume the online method they use is no worse or better than by telephone.
- The margin of error reported is based on the total of 801 responses and not on the smaller samples such as decided voters. They also do not report in the +- 0.5 percentage point margin of error due to the rounding of the results, this is relevant because they report their margin of error to the tenth of a point but not the actual results. If this poll follows the norm of 80-85% of the respondents having an opinion, the actual margin of error is 3.75 to 3.87 percentage points. Add in the rounding error and the actual margin of error on the headline results is more like 4.25 to 4.37 percentage points which is not only significantly higher than the reported 3.5 percentage points, it means a much flatter bell curve.
- The pollster does not indicate the size of the samples for the breakout groups, though from my calculations of it, it is very hard to see any breakout samples that are large enough to provide useful data. As an example the Vancouver Island numbers should have been from only about 140 respondents which is a margin of error of 8.28 percentage points, add to that the rounding error and it becomes 8.78. Providing very small samples with no indication of the margin of error is not responsible.
- They state the data is weighted to reflect the demographics of BC from the census. This increases the margin of error but that is not reported. I have no way to measure how much this changes the margin of error all I can say is that the more weighting there is the large the margin of error becomes.
- Weighting for census data also means the pollster is assuming voter turn out is not any different based on any demographics, this is a very flawed assumption. The older you are the more likely you are to vote, the richer you are the more likely you are to vote, and finally the more educated you are the more likely you are to vote. All indications are that older, more educated and wealthier voters are more likely to be on the right side of the spectrum. Weighting for the census demographics is likely overstating NDP and Green support but I have no data to show how much of an impact it would have in this poll.
- The pollster asks about how people who voted NDP or Liberal in 2009 would vote now but do not give us any idea of how many of the 801 respondents this represents. Based on a weighting that reflects actual voting in 2009, this would mean the largest sample was no more than 188 people. Any measure of current vote intent based on their 2009 vote would have a very large margin of error, 7.2 percentage points for the 2009 Liberal voters and 7.5 for the NDP. They do not ask who people who did not vote in 2009 are supporting. This should represent a weighted sample of close to 400 people.
- The Angus Reid polling series is out of step with the only real data we have, the by-elections earlier this year. In their March poll and the latest one the Liberals and Conservatives are nearly tied, in May the Liberals had a small lead. If this were true the BC Conservatives should have won in Chilliwack Hope and not come third. The by-elections point more to the BC Conservatives being in the mid teens in support and the Liberals nearer to 30%. The pollster does not address why their numbers are at odds with actual results.
- The attention to detail does not seem to be there in the poll. According to what I have been told, the pollster asked not about the BC Conservatives, but the BC Progressive Conservatives. Asking the wrong name must have some impact on the result but what I do not know.
 |
| Apparently a screen capture from the July Angus Reid poll |
10 comments:
with your track record, who cares what you think, you got it completely wrong in BC By-elections.
NDP is surging across Canada..Ipsos Reis..Angus Reid,,,,Forum research and Mustel are all saying the same thing.
Get over it Bernard, BC Liberals are dead and so are the Harper and Cummin`s Conservatives..
And all those BC Liberals fleeing the sinking ship should give you a clue, or do you need more bread crumbs.
Bernard,
Reid has extensive experience in both telephone and online surveying and weights with that experience in mind.
I think it pays off in the track record, which is pretty good - at least equal that of telephone surveyors.
The other thing is of course that every telephone surveyor weights as much or more. Seniors and young people answer telephone surveys less often than others. Men less than women etc...
The issue of the by-election is a red herring in my view. It's true that the conservatives under performed all their polling numbers in the by-elections, but there are more explanations for this than simply "the polling was wrong."
Turnout is probably a greater factor in the results and Get Out the Vote operations are key to turnout. For anyone remotely close to the campaigns it was clear that the Conservatives were nowhere on the ground and that cost them votes on election day. The biggest example of this was in the advance polls, which are entirely organization driven.
So my view of the by-elections is that the weak Conservative organization was responsible for their poor showing. If they fix that before the election their result will be more like their poll numbers. If they don't it will be more like the by-elections.
I think the most reasonable thing to say about the polling numbers is that they are all in the same ballpark, except for Ipsos, which has the Liberals too high and the NDP too low.
Regardless they all show a whopping big loss for the Liberals if fundamentals don't change.
I was largely wrong in the by-elections because I relied on all the pollsters and some other data points.
Other pollsters this spring have found the gap between the Liberals and the Conservatives be much closer to the by-election results.
If you read this blog regularly, you should know that I have been critical of pollsters for several years now and my critism is only increasing as a I see more and more systemic problems with most pollsters.
The only pollster I know of that is trying address the issue of problems in polling is Frank Graves of EKOS.
I have trouble taking Angus Reid seriously when they release such an unprofessional polling report. They do not show nearly enough information for anyone to understand much beyond their headline numbers, even there they do not release the margin of error numbers for the party support levels.
I should correct one thing...while in the past there has been a definite tendency for people with higher incomes to vote Socred/BC Liberal. Things are more mixed with older voters and there is evidence that rightwing parties (ie: the federal Conservatives and the BC Liberals) are losing the edge they once had with seniors - guess what? seniors don't like parties that tamper with pensions and health care! When it comes to people with university educations - they actually skew to the NDP and away from the BC Liberals. it makes sense - Adrian Dix comes across like a well-educated competent professional, while Christy Clark comes across like a ditzy cheerleader trying to pretend she's Sarah Palin. If you went to university you would probably be embarrassed to have her as your premier!
It should be noted that in the 2009 BC election ALL the telephone polls UNDER-estimated BC NDP support EXCEPT the Angus Reid online poll which was almost dead-on. So go figure.
In the 2009 election Angus Reid was consistently low for the BC Liberals and missed the mark in their final poll by a significant amount. They, like everyone else, over estimated the Greens.
I see no evidence to show Angus Reid is particularly accurate. All I can see is a tendency by them over estimate the NDP in polling. I am not sure if they are suffering from what Forum seems to be suffering from - over estimating the main opposition and under estimating the government.
I should added that I was less than impressed with the stunt they pulled in Ontario. The polled on Oct 3-4 and found the PCs leading by 3 points, the only pollster to do so.
They then released the results of a poll that was in the field on Oct 4-5 and found the opposite.
The very fact that they released two conflicting polls but did not explained why one of the polls was wrong and have not "shown their work".
The first poll had a much larger sample than the second one but it was significantly wrong
In 2009 Angus Reid's final poll on election eve had the BC NDP at 42% - and when the votes were counted the NDP got...drum roll please...42%. Where is the overestimate of the opposition???
As you may recall the BC Liberals won the popular vote over the NDP in 2009 by 3.5%. That final Angus Reid poll was ever so slightly slightly off giving the BC Libs a 2% lead, but look at how wildly off the other polls were: the final Ipsos poll had an 8 point Liberal lead, the final Mustel poll had a 9 point Liberal lead and the final Environics poll had an 11 point Liberal lead!
I see no evidence whatsoever that Angus Reid's track record is any worse than anyone else's and in fact - as far as BC is concerned - they seem to be the only ones who got it right last election.
http://en.wikipedia.org/wiki/British_Columbia_general_election,_2009#Opinion_polls
They had the Liberals too low in their last poll. The gap was 2 percentage points, not the 4 that happened on election day. Their last poll was not accurate because their results for 2 of the parties was far off of the results on election day.
The Angus Reid poll a week earlier had the NDP higher than other pollsters and the Liberals at their lowest in the campaign. They also had the Greens at 13%, a long way from 8.2%
Is Angus Reid worse than most? In general I do not think so but in the case of teh 2009 BC election there were not really enough polls by each of the firms to know who was doing well and who was not.
Currently we have 2 polling forms with the Liberals and Conservatives close to each other - Forum and Angus Reid. We have four firms that have the Liberals running 13 to 23 points ahead of the Conservatives - Ipsos Reid, Mustel, Justason and NRG. Justason is former Mustel associate Barbara Justason and this is the only poll I know of that she has done. NRG does not seem to have done a lot of political polls.
There seems to be issues with how the polls are conducted because we have two quite different polling patterns. I lean towards the second one being closer to reality because the by-elections more closely confirm those results.
One thing that jumped out at me from the ARS poll was the fact that the Conservatives were in 2nd place in Metro Vancouver. (Metro Vancouver typically representing 50% of the respondents in a BC opinion poll.
That's highly doubtful and doesn't make any sense from an "on the ground" perspective. Furthermore, that statistic was not corroborated by the Porty Moody/Chilliwack by-elections.
As for "Get Out the Vote" - I doubt that the NDP had any such organization in Quebec when it won the majority of its federal seats in 2011.
A balance of Ipsos, Mustel, and Justason seems much closer to today's BC political mark/reality than what ARS and Forum seems to apparently show. IMHO.
FWIW, even the provincial NDP was sceptical of a BC ARS poll back circa October, 2008 showing them in the lead over the governing Liberals.
The BC NDP publicly stated that it did not corroborate their own internal polling numbers and the ARS poll was also not corroborated by both Ipsos and Mustel at the time.
The polls I find least professional are the party internal ones, few people inside parties are willing call bullshit on bad work because the pollster is typically a supporter and a campaign strategist. There is no one neutral there to question the data
Post a Comment