When it comes to making predictions, Nate Silver is world-renowned as a fox, and he is very pleased about that. Unlike hedgehogs, who have only the one big idea that offers fragile predictions, foxes have many small ideas that enable them to deal more effectively with the risks and uncertainties of life. In Nate’s case, a supposed foxiness enabled him to successfully predict the outcome of all 50 states in the 2012 presidential election; a feat that set him on the road to becoming one of the world’s most respected statisticians and political forecasters. Times Magazine voted him one of the world’s 100 Most Influential People. The Independent has referred to him as the “Galileo of number-crunchers”. The Sunday Times thinks he is “the age’s Brunel”, and the Observer hails him as “a new kind of political superstar”. He has a list of accolades as long as my arm, and the Wikipedia entry for his FiveThirtyEight.com website, upon which his famed predictions were regularly posted, is impressively extensive. He is seen as one heavyweight, foxy dude who has made a living from putting hedgehog predictions to shame. So, when I came across his book, The Signal and the Noise: The Art and Science of Prediction, and saw that it had a full chapter on climate change, I just had to buy it.
In many ways I wish I hadn’t. I was looking forward to seeing someone with foxy talents and deep insights into the nature of risk and uncertainty challenging the climate scientists’ analytical approach. Instead, he concludes that there is nothing particularly wrong with how they do things. But that’s because his own insights into risk and uncertainty are not always what you might expect from the world’s foxiest fox, and much more like what you might expect from the world’s hoggiest hedgehog. It turns out that Nate Silver is actually a hedgehog who thinks he is a fox and has successfully convinced everyone else to accept his self-assessment. Well, nearly everyone.
A fox should be more inquisitive
I have a confession to make. So eager was I to enjoy the fireworks display that I went straight to the chapter on climate change before reading the rest of the book. Alas, the chapter turned out to be a damp squib. I’ll grant you that he talked to a lot of experts before committing to print, but not a lot came out of it.
For example, he sat down with a NASA scientist, Richard Rood, who told him there were three types of sceptic: the sort who is a political partisan with a self-protecting agenda; the sort who is ill-informed but naturally contrarian; and the true sceptic who is scientifically informed and works within the framework of the climate science consensus (Nate places considerable store in the importance of consensus. The existence of a scepticism that sees why he shouldn’t doesn’t make it onto the Rood list). So, apart from Nate accepting a simplistic taxonomy of climate scepticism, nothing of any value came out of this encounter.
He then sits down with Kerry Emmanuel, a MIT meteorologist, who injects some type-3 scepticism by pointing out that climate models are not bug-free (who knew?) and not all scientists fully agree on the accuracy of climate models in all respects. Nate seems to suggest that the existence of these hometown misgivings somehow renders all forms of external scepticism redundant. Still, no great insights into how risk and uncertainty are dealt with.
He sat down with Michael Mann and yet wrote up nothing more than the predictable complaint that climate scientists are under attack from bad actor deniers (“We are in a street fight with these people”). Nothing is said about Mann’s notoriety as — shall we say — an inventive statistician. He was also at pains to pass on the “pertinent fact” that the Climategate scientists were cleared by a panel of their peers. So, I think it is fair to say that investigative journalism isn’t Nate’s strongest suit. There’s certainly nothing insightful regarding the nature of risk and uncertainty, but remember that this is Michael Mann we are talking about here.
He sat down with Gavin Schmidt to receive a lecture on uncertainties in climate modelling. This was the perfect opportunity for Nate to raise the problematic way in which climate scientists insist on treating the spread of model ensemble outputs as if it were a probability distribution when it is, in fact, nothing of the sort. Gavin knows about this, so Nate would have found a sympathetic ear. But there’s nothing. It’s as if Nate isn’t a fox after all.
He then quotes William Nordhaus by trotting out the adage “uncertainty is not a reason not to act”, without delving into the ways in which it most certainly could be. So no, definitely no foxiness here.
As for the hot dispute within climate science between those supporting the storylines approach to risk assessment and the proponents of probabilistic Detection & Attribution methods – well, he doesn’t even go there.
I waited in vain for Nate to reveal his foxy skillset and dig deeper, but all I read was a pretty accepting albeit well-written summary of the standard narrative – just even-handed enough to give the impression that his was a fox’s open-minded analysis, but not so much so that he got into the chicken coop and did what foxes do.
A fox should avoid the simplistic
It was at this stage that I decided to read the book from the start. I was keen to find out what sort of strange fox it is that had no appetite for chicken blood.
The first inklings of a problem were to be found on page 29, where Nate, quite rightly, points out the importance of distinguishing between risk and uncertainty. He proceeds to offer definitions of the two, in which he says risk is something you can put a price on because the odds are calculable, whereas uncertainty is the risk that cannot be measured because there is not enough information to calculate the probabilities. Alarm bells sounded, not because I failed to recognise that this is one way in which people do indeed make the distinction, but because it is a distinction that glosses over a lot of important detail that a fox would not overlook. Specifically, a fox would not miss the fact that all risk is predicated upon uncertainty, and this is true whether or not the probabilities can be easily calculated. Consequently, people will refer to uncertainty irrespective of whether the probabilities are known. Even Nate himself does this on page 61 when he presents a well-formed probability distribution and states that “the wide distribution of outcomes represented the most honest expression of uncertainty”.
To be honest, ‘sloppy’ would be the kindest thing to say about his attempts to explain the basic concepts. But I had a deeper suspicion that his clumsiness suggested someone who, despite his undoubted fame and success, misconstrues the relationship between probability and uncertainty. I was also becoming increasingly concerned that he had yet to introduce the important distinction between variability and incertitude. Perhaps we are dealing here with a hedgehog who wouldn’t recognise a chicken if he saw one.
A fox with a big idea?
So, it was at this point that I decided to break off from reading his book and search around looking for the secret sauce behind his trademark tool (FiveThirtyEight). How did it produce the super-accurate predictions that had brought him so much public acclaim? I had the horrible suspicion that when looking under the bonnet I would find out that methods suited only for analysing variability, i.e. Monte Carlo (MC) sampling methods, were being used to analyse incertitude. My fears turned out to be well-founded.
In fact, FiveThirtyEight (from hereon, 538) used MC extensively. To put it simply, Nate built a probabilistic model of how likely voting would go in each state based primarily upon historical polls and their reliability, but he also included likely correlated errors between states. He would then run a simulation based upon a sampling of the model’s baseline probability distributions. This simulation would be repeated thousands of times, thereby generating a probabilistic average outcome. As new information became available, Bayesian updates were applied and the process was repeated to generate an updated assessment.
The problem, however, is that polling errors and systemic shifts in voting are primarily epistemic uncertainties (gaps in our knowledge), whereas MC simulations are fundamentally designed to model aleatoric uncertainty (inherent, irreducible randomness, like rolling a pair of dice). By using MC methods, Nate’s 538 was liable to overstate the confidence in its predictions because there was always the possibility of an epistemic uncertainty that was simply unaccounted for. Applying a tool built for pure randomness to solve a problem caused by imperfect data will always be a core vulnerability of the predictive model. It turns out that Monte Carlo is Nate’s single big idea – the hedgehog that ensures a fragile prediction.
Enter Mr Swan
Nate has responded to criticism of his use of MC in just the same way that all others who misuse it do – he simply says it is a powerful tool that offers a pragmatic compromise by converting epistemic uncertainty into a proxy of aleatoric randomness. He adds that volatility in successive predictions is to be expected as new information becomes available; it is not a sign that methodological weaknesses had rendered the initial prediction fragile.
Nassim Taleb of Black Swan fame disagrees. He noted that successive updated predictions made by 538 would often vary considerably, in stark contrast to the precision of each prediction. This, he points out, is a sign that something is structurally wrong with the predictive algorithm. Adding further information and re-running the prediction should not cause the predictions to wander off all over the place; a true probabilistic forecast for a distant event should be highly stable. To put it technically, in probability theory a succession of true forecasts should behave as a martingale, i.e. a sequence of estimates where the best guess for tomorrow’s value is anchored to today’s. The 538 succession of forecasts were not exhibiting that property because its MC algorithm makes a massive, unscientific assumption — the assumption that the future will not contain a type of epistemic ignorance that we haven’t seen in the past. To put it bluntly (as indeed Taleb has in online spats), 538 looks more scientific than it is, the reason being that it misconstrues the relationship between probability and uncertainty. Aha!
Given that Nate Silver is well-sold on the idea of converting epistemic uncertainty into a proxy of aleatoric randomness, it is no wonder that he should fail to challenge climate scientists who are playing the same dodgy trick with their model ensembles. In fact, if he did come across it in his tête-à-têtes with Schmidt et al, he likely would have simply nodded with approval, as one hedgehog to another. At the end of the day, Nate Silver is a highly acclaimed forecaster who seems to have a blind spot when it comes to the critical difference between variability and incertitude – or at least he chooses to ignore the distinction when selecting his tools. Which raises the question as to how he has managed to achieve his current position of fame and glory. How has Mr Hedgehog made such a successful career as a self-proclaimed Mr Fox? This question isn’t actually as difficult to answer as you might think. But we have to start by explaining how a fragile technique was so successful in 2012.
The making of a legend
Nate’s massive reputation was forged in the 2012 presidential election, where he correctly called the winner of all 50 states. This seemed a magical result, and so he was lauded as an infallible data wizard. However, the truth is that 2012 was a remarkably stable election with abundant, accurate polling. Any heavily data-driven forecast (and to be honest, Nate’s approach is nothing if not data-driven) would fare well given how clean the data was. Admittedly, Nate was very foxy when it came to ensuring that a variety of types of information were included in his analyses, but at the end of the day he just got lucky. The fragility of his big idea for data modelling (Monte Carlo Simulation) was present, but it wasn’t exposed because the epistemic uncertainty that year was exceptionally low. He happened to choose a year when converting epistemic uncertainty into a proxy of aleatoric randomness wasn’t such a dumb idea.
The rest of the story is a classic case of how the media fuels the Matthew effect. Every guest appearance on TV, every headline-grabbing update on 538, every fawning write-up from journalists entranced by his ‘science’ approach, just added to and reinforced the myth until his prowess could no longer be questioned. With the Matthew effect, it is important to get off to a good start, and the 2012 presidential election success certainly did that for Nate. And even when someone comes along, like Taleb, and demonstrates the mathematical flaws in Nate’s methodology, the bubble still doesn’t burst. Who is this who dares challenge the great wizard? The Nate Silver story is a sobering reminder that in the public sphere, a flawed approach that supports a confident narrative will almost always beat a rigorous approach that wears its limitations on its sleeve.
Nate Silver and the climate science community were made for each other: one a forecasting legend, the other a forecasting industry. Nate’s fame earned him an open door when it came to approaching the climate science luminaries, and the ensuing liaison benefitted both sides through a mutual turbo-charging of the Matthew effect. The scientists received the nod from the great man himself, and the great man himself further enhanced his standing by rubbing shoulders with the scientists. It was a win-win, but not amongst foxes. It was instead a lesson in what can be achieved by hedgehogs once they get to dictate the consensus. The hedgehogs end up winning, even when their big idea is ropey.
One day I’ll finish his book.
Feature image: Fivey Fox, the official mascot for FiveThirtyEight during the 2020 presidential election.
