Ontario by-elections and polling from Forum Research

Because polling has done rather badly over the last few years, I thought I would look at how well Forum Research did with their polling of the by-elections.

Overall their results from their final polls look better than I had expected, but they are still over sampling people aged 55 and older.   One result of this is that it is really questionable if one can call this a random sample of public opinion.   

Results that are reasonably close to the election results are in blue, ones that are not are in red.

Etobicoke Lakeshore - a sample of 565 decided voters
              Liberal   PC    NDP  Green  Others
By-elect result 42.0%  46.6%  7.8%  2.3%   1.3%
Forum July 30   43%    47%    7%    3%     1%
363 of the respondents were aged 55 or older - 59.6%.  They are only 34.7% of the adult population.
In total 1,952 people gave responses in one of the four Forum polls of which 1214 were 55 or older.  This means 3.4% of all the people over 55 and older answered a Forum poll during this by-election

That all noted, their results are very close here.

London West - 495 decided voter sample size
                 Liberal  PC    NDP  Green Freedom
By-elect result  15.8%  32.7% 41.9%  4.3%  5.0%
Forum July 31    15%    38%   36%    6%    5%
The sample had 337 respondents aged 55 or older, 68% of total respondents.  Only 35.7% of the adults are aged 55 or older.

Here they managed to correctly capture the Liberal support and the vote for Al Gretzky of the Freedom Party.   The results for the PCs and the NDP were outside of the 95% confidence interval.  They should only be this far off once in 50 to 70 polls
   
Ottawa South - 488 decided voter sample size
                Liberal  PC   NDP   Green
By-elect result 42.3%  38.7% 14.3%  3.1%
Forum July 31   36%    52%    9%    3% 
The sample had 345 of 488 respondents aged 55 and older which is 70.7% but they are only 33.1% of the electorate.

Here they utterly missed not getting anywhere close to the results for any of the three major parties.   The result is so far off for the PCs that this should not happen more often than once in 10,000 polls.   The numbers are wildly off

Scarborough Guildwood - 359 decided voter sample
               Liberal  PC    NDP   Green
By-elect result 35.8%  30.8% 28.4%  2.2%
Forum July 30   38%    31%   27%    3%
250 of 359 respondents were 55 or older - 70.4%.  They are actually 33.8% of the adult population

These results are within a reasonable range of the election results

Windsor Tecumseh - sample size of 292 decided voters
               Liberal PC    NDP   Green  Other 
By-elect result 11.9%  20.1% 61.3%  3.6%  3.1%
Forum July 31   12%    28%   52%    6%    2%
195 respondents were 55 or older - 66.8% who are only 38.0% of of the adult population.
While they got the winner right, the margin they missed the NDP and PC support levels by are way outside of reasonable.  They should only be this far off in less than one poll out of 100.

So two of the polls are within a reasonable range and the other three are no where close to a reasonable reflection of public opinion.  In one case it is so far that as a result it should never occur in a proper random sample survey of the public.

Two out of five as reasonably accurate is not good enough and it means we can not realistically know which results are going to close and which ones are not.   I do think there is a major methodological problem with Forum because of the very high response rates they get from people aged 55 and older.   It shows us that their sampling system is not creating a true random sample of the public.   Why are they missed younger people?