Weekly Writ 11/13: How the Canadian polls did this fall
Assessing the accuracy of the polls in three provincial elections.
Welcome to the Weekly Writ, a round-up of the latest federal and provincial polls, election news and political history that lands in your inbox every Wednesday morning.
This newsletter might be too long for your email inbox — if so, at the very bottom you should be able to click to expand it to read it all!
Now that the results in Saskatchewan and British Columbia have been finalized, we can take a look at how the polls did in the trio of provincial elections held in October.
We’ll start in British Columbia, where the polls did a good job at capturing the close race between the New Democrats and Conservatives. For the most part, they also accurately showed that the NDP had a narrow advantage.
All the pollsters in the field during the last week of the campaign had their results fall well within the margin of error. The NDP took 45% of the vote, and every pollster was within three points of that score. All but one were within two points. There was only a marginal under-estimation of NDP support, as the party was averaging 44% in polls conducted in the last week of the campaign.
Every pollster in the field in the final week was within two points of the Conservatives’ score of 43%. The polls were bang-on for the Conservatives, as they averaged 43% in the final week.
The Greens were a little harder to pin down, as every pollster over-estimated them. This is explained by the fact that the Greens didn’t have a candidate in about a quarter of ridings, and pollsters would have been nearly perfect had they all simply reduced their Green score by 75%.
Pallas Data and Léger were the best-performing pollsters, but the difference between them and those lower down the table was marginal. Both were off for the three major parties by a point or less, while the per-party error was two points or less for Forum Research, Ipsos and Liaison Strategies.
All-in-all, a strong performance across the board, including at the regional level. Polls correctly pinpointed the Conservatives’ lead in the Interior and the NDP’s lead on Vancouver Island and showed the Lower Mainland was a close race that leaned NDP.
That the lowest-ranked pollsters were out of the field earliest shows how it’s important to poll as close to election day as possible.
Only two pollsters were active in New Brunswick at the provincewide level, and both were in the field in the last days of the campaign. While both under-estimated the size of the Liberal victory, the two pollsters put up a respectable performance that forecast the Liberals’ majority government.
Mainstreet was on the mark for the Liberals, but over-estimated the PCs by three points and under-estimated the Greens by four points. Forum Research was off by one to three points for each party. This was a rare election in which the Greens were under-estimated by the polls.
While B.C. and New Brunswick were good elections for the pollsters, Saskatchewan was not. The top two finishers were polling more than two weeks, and in one case nearly a month, before election day. The four pollsters who were in the field in the last week under-estimated the Saskatchewan Party by an average of six points and over-estimated the NDP by nine points. That net miss of 15 points is enormous and a clear failure on the part of the polls.
Cardinal Research and Janet Brown Opinion Research correctly put the Sask. Party in the lead, though their margin in the polling was half the size of their victory margin in reality. Both polling firms were also in the field some time before election day, with Brown’s last day in the field being 25 days before the election was over. Cardinal polled for a full 15 days, straddling both the early and the late days of the campaign. Nevertheless, they were closest to the mark and win the Saskatchewan sweepstakes.
Once again, pollsters struggled with Saskatchewan. Their average error for the two major parties was 6.7 points per party, up from 4.4 per party in 2020. That election was also a polling miss, but since the Sask. Party’s margin of victory was so huge it was an inconsequential error.
The polling miss here was across the province, but it was concentrated outside of the two cities. On average, polls over-estimated the NDP by four points in the urban centres and eight points in the rest of the province. They under-estimated the Sask. Party by two points in Regina, four points in Saskatoon and six points in the rural areas. While those urban performances aren’t great, clearly there is a particular issue with polling the rural areas in Saskatchewan.
A special mention should be made for Research Co., however. Their provincewide poll was no better than those conducted by other firms in the last days of the campaign, but their regional results were outstanding. Research averaged an error of 0.9 points per party at the regional level. Cardinal had an average regional error of 2.4 points per party, while the other pollsters active at the end of the campaign had an average regional error of around six points per party.
Why Research’s good regional numbers didn’t add up to a good provincial score is puzzling. The cities were apparently carrying too much weight in Research Co.’s poll. Had it been weighted correctly, it would have been the best performer by far.
Any polling lessons from this trio of elections? I think the main one is a reminder that polling in each campaign is unique. There was no uniform under- or over-estimation of the right or left across these three elections. We can’t assume that the polls will miss high or low on a party based solely on where it sits on the political spectrum.
Now, to what is in this week’s instalment of the Weekly Writ:
News on the tough byelection test the Liberals face next month, the final results in B.C., an expanding map in Alberta and the receipts from two Ontario byelections. Plus, an update to the Prediction Contest.
Polls are holding steady across the country, while the provincial PCs are leading in both Nova Scotia and Ontario.
Big majorities for Tim Houston and Doug Ford if the elections were held today.
The Alberta Liberals phone it in for the #EveryElectionProject.
A milestone for Canada’s longest serving elected party leader.