One more newsletter for you before we publish our final FLIPR forecast shortly after midnight. If you’re new here, we do not usually post this much: it’s typically ~3 items a week, not three a day. But we hope you’ll join us for our final flurry of coverage, including our Election Night livestream and live blog tomorrow.
And we have a bonus episode of Still Counting out:
Reviews, comments, likes and subscribes are always greatly appreciated. Audio version here:
We covered a lot of ground on the show. But I wrote about those NYT polls and Haley Stevens yesterday, so I’m going to use this as an excuse to cover another topic that Galen, Clare and I were only able to scratch the surface of. I thought it would be nice to write about this ahead of time so it doesn’t feel like spin if a lot of FLIPR’s forecasts don’t come true.
I’ll sometimes get the question, “How do you know if a probabilistic forecast is ever really wrong?” which is an entirely reasonable thing to ask. In fact, let me give you a four-part answer. It’s certainly not quite as simple as saying “30 percent chances are supposed to happen 30 percent of the time.” That’s true insofar as it goes, but it’s incomplete. And if you’re being Bayesian about it, you can’t always use this as a get-of-of-jail-free card. A 0.03 percent chance happening is much more suspect for your model than a 30 percent hitting.
First, the canonical way to answer this. You want your probabilistic forecasts to be well-calibrated over a sufficiently large sample of true out-of-sample predictions. In other words, if you’ve forecasted 100 events that you said had a 30 percent probability of happening, and 27 of them or 34 of them actually did “hit”, your model is doing well.
The “true out-of-sample predictions” part of the answer is important. Backtesting doesn’t count. Any halfway-competently-designed model should fit the past data well; that’s the easy part. But the model may be misfit or overfit, keying off of essentially random factoids in past data that have no true predictive power and aren’t likely to repeat themselves. You may have noticed that in documenting our models, we tend to not make claims like “ELWAY would have gotten the exact score right in the Super Bowl XVI” or “FLIPR would have nailed the 1994 midterm.” Who cares. We already know what happened. I generally think it’s a negative sign when forecasters brag about their backtested results instead of explaining why they think their models make structural sense.
In the case of the Silver Bulletin/FiveThirtyEight lineage of models, however, we have something fairly unique: an 18-year-track record of making true out-of-sample predictions in advance of elections and sporting events, which were published ahead of time, and indeed have received a ton of public scrutiny. At FiveThirtyEight, we actually had an interactive that diligently tracked our forecasts (although the Internet Archive version of the interactive is a little hinky and Disney nuked the original version). Our sports forecasts were extremely well-calibrated: over a large sample size, the 30 percent chances really did hit about 30 percent of the time, and so on. Our politics forecasts were pretty well-calibrated too, although actually a bit underconfident: there were actually too few upsets relative to what our models expected. It’s hard to know whether this is a statistically significant differential, but it makes me a bit less worried when FLIPR is more confident about a race than the consensus.
A second way is a very Nate way that some of you won’t care for. Would the model have made you money if you’d been betting on it — relative to other models, the conventional wisdom as far as it can be sussed out, or betting markets? If you trusted our model in 2016, you should actually have made a large bet on Trump, because you could get roughly 5-1 or 6-1 odds when our model thought he’d win around 1 time in 3 instead.
We could get into a more detailed discussion about things like Brier scores — you not only want your model to be well-calibrated, you also want it to discriminate clearly between relatively more and relatively less likely outcomes. You’ll be calibrated perfectly if you always forecast every race as 50/50 but the model doesn’t tell you anything.
But I actually think some of the most important ways to evaluate a model are qualitative.
The third thing I always look at is whether a model responds proportionally to new evidence. If either the magnitude or certainly the direction of changes in our model doesn’t pass a common-sense test in accordance with the new evidence, that’s often a sign it’s poorly constructed.
For instance, when that set of very strong NYT/Siena polls for Democrats came out last night, their win probability in the Senate improved by about 2 percentage points in FLIPR. That seems fairly rational. Our odds for Democrats were already pretty favorable. And the NYT/Siena polls have generally been on the higher side of the averages for Dems, so FLIPR was expecting some relatively good numbers. Still, they’re a highly rated pollster, and their polls weren’t just good for Democrats but great in some states like Michigan, often showing favorable trendlines.
By the way, you can also kind of combine #2 and #3. Here, for instance, are Democrats’ Senate odds this year in FLIPR as compared to Polymarket:
This doesn’t always happen, but FLIPR has been leading Polymarket this year, you might notice. They’re going to wind up in roughly the same place, but FLIPR got there faster. Although I know this is self-serving, I’d argue that’s already a little bit of a “W” for FLIPR. (You could, in principle, have bought Democratic positions and sold/hedged them now to lock in a profit.)
Fourth and finally, you should make some effort to determine whether your model was lucky or good. This is sometimes easier in sports. With the Denver Broncos under Bo Nix, for instance, there have been some cases where ELWAY won its bet, but the team really had no business winning the game.
FLIPR has various hypotheses. Just to take a random few:
-
The incumbency advantage matters less than it once did.
-
State-by-state polling errors are correlated in midterms but not as much as in presidential years.
-
Gubernatorial races are less partisan than Congressional ones and produce more crossover voting.
Testable evidence for all of this exists in any given set of election results. If candidate quality matters less than FLIPR expects, maybe Ken Paxton and Roger Marshall hold on. If it matters more, though, maybe they lose but Susan Collins wins.
For all of this, by the way, margins ought to matter and not just the final calls. We’re quite skeptical about Democrats’ chances in Louisiana, for instance, but if Jamie Davis loses by just 1 point, that’s probably a bad forecast and not a good one even though our “call” will be right.
Conversely, some GOP Senate wins would be much worse for FLIPR than others. If Democrats pick up three Senate seats but lose recounts in Texas and Alaska, that’s much closer to FLIPR’s central distribution than if Roy Cooper loses North Carolina.
What sort of things will we be looking to evaluate and potentially change in FLIPR and our other election models? Here are some things I’ve at least thought about. I always think it’s helpful to keep a mental list of these ahead of time so you’re not too overly driven by results and wind up fighting the last war.
-
Whether to treat polling errors as being somewhat persistent, e.g., Susan Collins should get a boost because she beat her polls last time around.
-
As discussed in SBSQ, whether to apply some sort of fancier adjustment for undecided voters, i.e., undecideds are presumed to be blue in Minnesota but red in Kansas.
-
Whether to account for WAR or past election margins for non-incumbents; currently we’re only doing this for incumbents.
-
Whether to include sources of fundraising other than individual contributions. Currently, we only include individual contributions and not things like PAC money. The individual dollars have been very lopsided in favor of Democrats this year, while the corporate and VIP money has gone the other way.
-
Whether there’s some feasible way to account for candidate ideology in our “fundamentals” ratings. Currently, we only do this for incumbents, and it’s not really based on ideology per se but rather party loyalty in Congressional voting records.
-
Whether improvements could be made in our statewide partisanship scores, like accounting for the differences between midterm and presidential electorates, or changes in voter registration or migration patterns.
-
Whether to remove the expert ratings layer from FLIPR Deluxe.
-
Whether to force our estimate of the national House popular vote to more closely match our generic ballot numbers — currently, there’s a difference of about 1 or 1.5 points between these. This is intentional, not a bug, but it’s something we might examine if Democrats considerably outperform their popular vote forecast.
-
Whether to account for early voting somehow.
-
And this would be the big one if polls have a bad night: whether generally to de-emphasize polling in favor of other factors, and how that would be implemented, exactly.
But in each case, there’s at least some reason that we aren’t doing these things already. #5 and #6 would be nice, and probably helpful, but it’s not necessarily easy to find the right data, especially if we also need it for lots of past elections to help validate the model. I have a fairly strong prior against #1. For #4, the individual contributions data is nice to have as one of the relatively few objective non-polling inputs in FLIPR. Incorporating early-voting data (#9) would be a beast. It’s a moving target, and you’d need a lot of state-by-state customization; it might take months to do right.
Still, I do try to look at whether the sort of reasonable changes we could make but haven’t made systematically tend to favor one party or the other. Changes #1, #2, #4 and #10 would probably help Republicans this year. But changes #3, #5, #8 and #9 would likely help Democrats. It looks pretty even to me. I was slightly more worried when our model was more D than the consensus, but now prediction markets and other models have mostly converged toward FLIPR.
Of course, #10 is arguably as important as the other nine combined. FLIPR has actually moved slightly in the direction of being more fundamentals-based given the changes we made this summer. But it’s a bit truer to the polls than some of our competitors’ models, especially in estimating the national environment. However, models that make stronger assumptions about fundamentals and let them override polls are easy to overfit, and frankly have a middling-to-poor track record. Nor is it necessarily clear that this would help Republicans anyway, so maybe I was too quick to put #10 in their bucket. As I’ll argue in our closing story, there is lots of non-polling evidence supporting a strong year for Democrats too.
-
Join us for our Election Night coverage!
-
And so you know, we are updating ELWAY, it just got bumped down the homepage with all of our politics coverage this week.