A Tale of Two Angus Reid Polls

The polling firm Angus Reid has managed to do the most amazing thing in the Ontario election, they will likely to be the first polling firm to be able to claim they got the election results right and wrong.

Angus Reid released what various people told me would be their last poll in the election on October 4th

The headline from October 4th:

Tories Edging Liberals But Ontario Race Could Turn in Final Hours
The Progressive Conservatives keep the largest number of committed voters, but strategic voting could end up hindering their chances.

Then yesterday they released another poll, here is the headline for that one:

Committed Voters Push Liberals to First Place as Ontario Ballot Looms
Support for Progressive Conservatives collapses as fewer Grit voters in 2007 are considering to cast a ballot for the Tory candidate in their riding.

Here are the headline results for the two polls

• Party    Oct 4th Oct 5th
• Liberals   33      37
• PCs        36      33
• NDP        26      26
• Greens      3       5

I find the two headlines completely misleading when looking at their results.   The statistical margin of error for a 95% confidence level was 2.1% on the 4th and 3.6% on the 5th.   To me the results say that the October 4th poll was too high for the PCs and too low for the Liberals and then on the fifth it was reversed.  The headlines results are actually not far off each other, but the first one looks like it is far off from the rest of the pack in the election.

Releasing polls close together is normally only done by companies that have been doing daily tracking polls.   In Ontario in 2011 that has Nanos with EKOS coming on board at the end.   Angus Reid has never had a history of releasing two polls this close together.

The cynical person inside me says that the company went back into the field on October 4th when it saw that the polling numbers from October 4th poll were not close to all their competitors.    Maybe they always planned on a poll to be released on October 5th, but rarely have I seen a company collecting data on the day they release a poll which is what they were doing with the second poll.

Maybe a headline like "We released a last minute poll to hedge our bets and make sure we are close out competitors in predicting the election outcome."  would have been a better one to choose?

Given the stock Angus Reid has been putting in their electoral prediction record over the last four years, I suspect there was a very, very strong incentive to get a poll that would be close to the final result.

In looking over their document on how well they do with election predictions, they have written it up much of in ways that seem to be intended to mislead.   As an example, they state in the May 2009 BC election the BC Conservatives got 4% of the vote when in fact they only got 2.1% - they may have meant to say all others.   They also compare themselves to at least one poll that was released a week before the election.  Their track record in the document is ok but not stellar.

The document would be much more interesting if they could explain why they got some elections right and not others and could explain why other pollsters fared better or worse than them.   There needs to be someone to do a serious analysis of the accuracy polling and try to quantify the real sources of error within polls and get away from the statistical margin of error.  Seems like a good PhD project for someone in PoliSci in Canada if they could get a lot of detailed raw data from past polling in Canadian elections.

So which poll will they count as being accurate in Ontario?  The larger one released on October 4th or the smaller one released on October 5th?   Will Angus Reid admit that their October 5th poll was wrong, or at least their headline and analysis were wrong?

I really do not like that Angus Reid simply reports the numbers without saying how large they are.   As an example on page 4 of the October 5th released poll we get the regional results for Ontario.  Six regions are listed, though no clear boundaries of what the names mean.   The poll only had 1,003 respondents and only 747 certain to vote respondents.   This means the regional results are very, very small.   When it comes northern Ontario, it could have been as little as 40-50 people.    The implication of this is that the poll has 78% of people in northern Ontario voting for the NDP.  This is a result that everyone knows is wrong but has been released in the poll as if it has some connection to an accurate reflection of popular opinion.    A day earlier Angus Reid had the NDP at a lowly 39% in northern Ontario.

The lack of details within the poll of such things as did the person vote in 2007 and how, how likely are they to vote, how many respondents in each break out result, and something as simple as how many non-voters makes it very hard to for me to see any relevance to the election from the data released in the two Angus Reid polls.

There is no shortage of us political number crunching geeks out there that would parse the data very effectively if given the chance with the raw data.   The company in Canada at the moment that provides the most comprehensive data is EKOS.

I am critical of pollsters are the best of times, but in this case Angus Reid is really pissing me off.