This week, I’m pleased to update The Writ’s projection model with some major changes made to how it simulates uncertainty.
I know, big stuff.
But since the launch of the model last spring, I’ve grappled with two separate issues that became bigger problems in the Quebec and, to a lesser extent, the B.C. provincial elections. It’s not that the model wasn’t working well — it actually worked very well in projecting the outcome of those elections, making the right call in 91.4% of ridings in both provinces (and calling 96.8% of ridings correctly or, when making the wrong call, classifying those misses as tosses).
But it wasn’t expressing the uncertainty in those projections very well. The model was too uncertain, with too-wide seat projection ranges and too-high probabilities of unlikely events occurring, like a CAQ upset win in Quebec.
I won’t bog you down in the details, but in short the problem was that the probabilistic aspect of the old model was abstracted in a way that simply wasn’t working. While it kept the model simple (and running quickly) it was not the right approach and it didn’t make it possible to make probabilistic forecasts for each individual riding. And, as we saw in Quebec in particular, it gave too much weight to the extreme outcomes, and not enough to the most likely ones.
So, for the last few weeks I’ve been redesigning the probabilistic aspect of the model. The uniform/proportional swing model, the adjustments made for special cases based on past precedent and the vote projection model hasn’t changed — those were working well and have proven themselves so far in two provincial elections. But using those projections to then calculate the probability of various outcomes has changed entirely.
In the new model, simulations are run for each individual riding. For each party in each riding, randomized results are generated using a normal distribution, with the model’s root mean square error (calculated from tests from previous elections) used to determine the degree of variation from each party’s originally-projected support. To ensure that the model doesn’t treat every riding as an independent contest, the randomized results for each party in each riding uses a combination of a region-wide and a local error.
So, for example, in one election simulation the Liberals might be assigned an under-performance of two points across all of Ontario. That single simulation will then have the Liberals be two points lower than the projected result in every riding across the province, but there will also be a second level of local error for each riding to take into account localized effects. So, the Liberals might over-perform by two points in one riding in Ontario and under-perform by three points in another one, but that under- and over-performance is, for that single election simulation, compared to the two-point under-performance that has been applied throughout the province.
This is repeated 5,000 times, with election results simulated across all 343 ridings, to determine the winner of each seat in each of the 5,000 simulated elections, but also the total number of seats won by each party across the country in each of those individually-simulated elections.
This results in a truly probabilistic estimation of the odds of each party winning each individual riding, the likelihood of each party winning X number of seats, and the likelihood of each party winning the election. While this is not an innovation in the election forecasting world, it is something new for my model. Previously, the model’s simulations were for the country (or province) as a whole, simulating elections where a party wins a number of seats between their ceiling and floor, but not simulating which of these seats were actually being won or lost. This had the unintended effect of too many simulations resulting in parties virtually sweeping all the seats they had even a small chance of winning, or losing all the ones in which they had even a small chance of losing. The “tails” of the projection, so to speak, were too long and (often) too thick.
Switching to this new model should resolve the two issues that popped up in the Quebec and B.C. elections, and were also bugging me when it came to the federal model.
Firstly, the 95% confidence intervals for all parties is now far more realistic and far more useful.
The final projection for the Quebec election had the PQ between 32 and 68 seats and the CAQ between zero and 39 seats (the other parties’ ranges were narrower, simply because of their more limited number of contested seats). Those ranges represented nearly a third of the entire electoral map. That range was too wide to give people a useful understanding of the race, and the notion that the CAQ could win up to 39 seats was indefensible. The old model gave too much weight to extreme outcomes, which would’ve required the polls being off by, say, a net 10 points between the PQ and CAQ. Not impossible, but not really a high-probability event.
Using this probability model but making no other changes to the final projection, the PQ would have been projected to win between 48 and 75 seats and the CAQ between 0 and 12, a far more reasonable range. The Liberals would have been pegged at between 32 and 42 seats, the Conservatives between 11 and 29 seats and Québec Solidaire between five and 10 seats. The actual result fell well within these ranges.
In British Columbia, the final projection had the Conservatives winning between 29 and 75 seats and the NDP between 16 and 62 seats. Again, those ranges were simply too wide to be helpful. With this new model, the range would’ve been 40 to 71 seats for the Conservatives and 20 to 51 seats for the NDP, a reasonable reflection that a polling error could flip the outcome but that the Conservatives were the heavy favourites.
The new federal model now has the Liberals winning between 208 and 247 seats and the Conservatives between 59 and 95 seats. That’s still a wide range, but it is a reasonable one and far more informative than the last projection with the old model, which had the Liberals between 169 and 272 seats and the Conservatives between 46 and 147 seats. With the polls where they are, it isn’t plausible to suggest even an outside chance that an election held tomorrow would result in nearly the same seat count as in 2025.
This also has an impact on the odds given for a majority or minority. The final projection in Quebec would have been a 55% chance for a PQ minority and a 45% chance of a PQ majority, with no chance of another party winning. With the benefit of hindsight that’s a good call, but the old model was still giving a 3% chance of a CAQ win in the final projection (more than the Liberals’ 1%), which I knew was not realistic.
In B.C., the odds of a Conservative victory would’ve increased from 81% to 91% and the odds of an NDP win dropped from 18% to 9%. Considering that no poll had the B.C. NDP leading in the final weeks of the campaign, and most had them trailing by about five points, it was not realistic to still give the NDP a one-in-five chance of winning. A one-in-10 chance, though, considering the potential for a polling miss and some surprising local races? That would’ve been plausible.
The second issue this model resolves is the difference between the Median Projection (previously the Average Projection) and the Seat Projection. While the Seat Projection did not try to guess in which direction the polls would miss, the Median Projection used the probability model that did try to guess at the direction of the error. This is why the ranges could be extremely lopsided and why the Average/Median Projection could be very different from the Seat Projection. It was something I had to repeatedly explain and I think it was confusing to many readers.
Worse, I was committing one of the projection sins I had always tried to avoid. When a model tries to guess by how much and in which direction the polls will be off, it risks doubling its error. If the model guesses right about the direction of a polling miss, it will work very well. If the model guesses wrong, it will perform even worse than if it didn’t try to make that guess in the first place.
As pollsters continually update their methods to improve their accuracy, it’s risky to assume that errors the polls made in the past will be replicated in the future. We saw this in the U.S. midterms, for example. The conventional wisdom based on past performance is that the polls under-estimate Republicans. Instead, the Democrats beat their polls, and by a significant margin in some states. Any U.S. projection model that forecasted a Republican over-performance made bigger errors than models that simply stuck to what the polls were showing.
The folly of this was also shown (to a lesser extent) in both Quebec and B.C. Based on past precedent, the model expected the CAQ and the B.C. NDP to beat their polls. Instead, they under-performed the polls. It expected the Quebec Conservatives and Liberals to under-perform their polls. Instead, they beat their polls. In these cases, the model was better off not trying to guess at the direction of the error, especially when we consider that the sample size of past elections is so low to begin with.
But polling error is something that still needs to be taken into account in some fashion. Using the root square mean error rather than the standard deviation from the mean increases the uncertainty for those parties that have significantly under- or over-performed their polls without making a guess at how the polls will miss in the future. For the federal Conservatives in Alberta, for example, this no longer means assuming almost as a given that they will beat their polls by a significant amount. But it does mean increasing the error bars for the Conservatives in Alberta more than they had been before.
Now that the model doesn’t try to guess at the direction of a polling error, the Median Projection is no longer very different from the Seat Projection. I am continuing to present the Median Projection, however, because it still serves a purpose. If a party is leading in a lot of toss-ups, the Median Projection will be a little lower than the Seat Projection because that party will lose a disproportionate number of those toss-ups across the 5,000 simulations. Similarly, if a party is trailing in a lot of toss-ups, a disproportionate number of them will fall their way in the simulations. The number of safe ridings each party has will also play a role here — a party that is leading in only safe seats will have a lower floor than a party that is leading in only close ridings. The Median Projection will continue to serve as a bit of a guide for understanding the most likely outcomes, but the Seat Projection remains The Writ’s official call.
Now that abstraction has been removed from the probability calculations, it’s possible to be far more detailed in what the model is spitting out. You’ll see on the projections page that I have now included a table showing the high and low seat ranges for each party in each region or province of the country. I have also now included the probability of each party winning each individual riding. You can check those out at The Writ’s (re-named) Canadian Election Projections page. The new probability projections per riding are at the very bottom.
For the six byelections scheduled for December 7, that means I can now give probabilities of victory for each party in each seat. The Liberals have a greater than 99% chance of winning Scarborough North and Laurier—Sainte-Marie. The Bloc has an 80% chance of winning Saint-Hyacinthe—Bagot—Acton (the other 20% chance going to Liberals). The NDP has a 74% chance of winning Rosemont—La Petite-Patrie, with the Liberals have a 24% chance and the Bloc a less than 5% chance. (My gut call gives the Liberals a better chance than that, but we’ll see.) The Liberals have a 69% chance of winning Brantford—Brant South (31% to the Conservatives) and the Conservatives have a greater than 99% chance of winning Yorkton—Melville.
It also means I can produce fun graphics like this one, showing the results of all the simulations in Rosemont—La Petite-Patrie:
I will likely use these charts more often in the future. As you can see, the NDP is leading in the projection but there is a far amount of overlap with the Liberals (and a very small amount with the Bloc Québécois).
You’ll notice that the probabilities don’t show a 100% chance for any party in any riding. Even though many ridings have only one party winning the seat across the 5,000 simulations, the model is tested on only a limited number of real elections so it seems inappropriate to say that any electoral outcome has a 100% probability. When a party wins between 99% and 100% of the simulations in a riding, I show the probability of victory as >99%. When a party wins less than 1% of simulations (but more than 0%), I show that as >1%. To also de-emphasize the certainty of the model at the upper extremes, I am also showing a >95% chance of victory for parties that win between 95% and 99% of simulations, and a <5% chance of victory for parties that win between 1% and 5%.
To further simplify things, I’ve also done away with the “Close” and “Edge” calls. Now, a seat is deemed “Safe” if a party wins 100% of simulations and “Likely” if it wins more than 95% but less than 100%. A “Lean” call is for parties that win between 70% and 95% of simulations and a “Toss” is for parties that win less than 70% of the time, but more than any party. A party could be projected to be the favourite with victories in fewer than 50% of simulations in three-way or even four-way races. I’m glad that the model can now quantify and show that.
As I said when I initially launched The Writ’s new model earlier this year, it will be an iterative and evolving process. Already, some tweaks and adjustments have been made to improve the model. (Some of those minor tweaks are explained in the footnotes at the bottom of the methodology). I was very pleased with how it performed at the riding-by-riding level in both the Quebec and B.C. elections, and the vote projections also performed well, with a total error that was smaller than all but the best polls in each of those campaigns. But I wasn’t pleased with how it presented uncertainty. Now, I think I have a good way forward that will also open up some new analytical possibilities.
The next steps are to use this new foundation to launch new models. Coming soon will be a re-launch of the Quebec projection (it’s a minority government, after all) and new launches of models for Alberta and Manitoba, which are holding elections next year, as well as for Prince Edward Island — though the lack of polls out of PEI raises some question marks about how I should approach that one. And then I’ll roll out models for the remaining provinces.
Lots to do! But there’s a high probability that it’ll be fun.