Seems, madam? Nay, it is; I know not ‘seems’.

Hamlet (Act 2, Scene 2)

There are two — and only two — causes of uncertainty: the variability to be found in the real world and the incertitude that results from having gaps in one’s knowledge.¹ The two causes may act in isolation or co-exist. And that is all there is to it. Apart from that, the only other thing to keep in mind is that uncertainty caused by variability has to be treated differently to uncertainty caused by incertitude. If you fail to respect that distinction then you may incorrectly evaluate the uncertainty, and when critical decisions are being made under uncertainty, incorrect evaluations matter – they matter a great deal.

In climate science, both causes of uncertainty are relevant. The climate system is inherently variable (aleatory), and our understanding of that variability is subject to incertitude. Climate scientists therefore have to be careful in isolating the two causes of uncertainty in order that they may be dealt with appropriately. Ay, but there’s the rub — they are not in the least bit careful. In fact, there are times when they act as though all uncertainty is down to variability; it is as though a subjective opinion is interpreted as an observation of reality. I have touched on this subject before, but it strikes me that I have never taken the time to detail just how extensive and damaging this methodological eccentricity is. In fact, there is barely an area of climate science that doesn’t fall foul of this most basic confusion at one stage or another. And the worst thing is that climate scientists often know perfectly well that they are doing wrong, but they simply dismiss their wrong-doing as a pragmatic necessity. I hope to convince you that this attitude may be understandable but it is, nevertheless, fundamentally antagonistic to scientific integrity. If after reading what follows you are still happy with the state of affairs within climate science, then I have to say that you are a lot more forgiving than I.

The name of the beast

O, God! a beast, that wants discourse of reason.

Hamlet (Act 1, Scene 2)

If you haven’t guessed already, the problem I refer to is the extensive and thoughtless application of Monte Carlo (MC) methods. Monte Carlo is in fact a brilliant way of dealing with uncertainty caused by variability, but it cannot — indeed it must not — be used to tackle uncertainty caused by incertitude. But climate scientists do so all the time; so much so that I fear the following catalogue of indiscretion barely scratches the surface. I’ll treat each example in turn, although I think you will find that there is a great deal of overlap. I’ll start with the problem that I have written about before now, namely the mishandling of climate model ensemble uncertainty, but this time I will not be ending there. Things will get quite technical at times, so if you are looking for a quick read, you might want to skip to the summary.

Modelling is not measurement

And to the manner born, it is a custom more honour’d in the breach than the observance.

Hamlet (Act 1, Scene 4)

Let us say that you have a physical system that behaves unpredictably due to the presence of a number of parameters that are varying in value according to a probability distribution. Such complexity will mean that the outcome in any given instance will be uncertain, but it will be bounded. To get a feel for this, one can create a set of values for the system’s parameters based upon sampling of their respective probability distributions and then calculate the outcome given that set of sampled values. To get a feel for the most likely outcome, together with its uncertainty, one can repeat that sampling process thousands of times to produce an average and spread for the outcome. That is basically what Monte Carlo sampling does for you. It works because one is dealing with the stochastics of the real world, in which well-defined probability distributions are a portrayal of variability.

Now let’s say we are dealing with a system for which the parameter values are simply unknown due to a lack of data (they are not parameters that are varying, they are fixed but unknown). In fact, let’s say we are also dealing with a system for which the understanding of the physics causing any variability is subject to incertitude. Sure, you can do the Monte Carlo sampling trick to generate a most likely outcome, but upon what basis do we choose the probability distributions from which to sample? One way is to ask a number of experts to offer their guess for the parameter values and then sample from the distributions formed by their set of guesses. But that is not the same thing. We are no longer analysing the real world in its variability; we are analysing the world of expert incertitude represented by their models of the real world. And modelling is not measurement. Subsequently, the outturn curve produced by the Monte Carlo simulation cannot be treated as an objective calculation akin to a probability distribution since expert incertitude is driving the uncertainty, not real-world variability.

This confusion between variability and incertitude becomes even more stark when ensembles are created from models that do not share the same structure, as in CMIP6. Each model represents an expert group’s understanding of the Earth’s climate system, complete with parametric and structural uncertainty. Taking a set of such models and treating each model’s projection as a plot on a probability distribution is only valid when the ensemble represents a random sample of all possible models, which will not be the case. Professor Eric Winsberg of the University of South Florida explains the situation as follows:

Ensemble methods assume that, in some relevant respect, the set of available models represent something like a sample of independent draws from the space of possible model structures. This is surely the greatest problem with ensemble statistical methods. The average and standard deviation of a set of trials is only meaningful if those trials represent a random sample of independent draws from the relevant space—in this case the space of possible model structures. Many commentators have noted that this assumption is not met by the set of climate models on the market…Perhaps we are meant to assume, instead, that the existing models are randomly distributed around the ideal model, in some kind of normal distribution, on analogy to measurement theory. But modeling isn’t measurement, and so there is very little reason to think this assumption holds.

So, this is a problem that is well-known amongst climate scientists. Gavin Schmidt, the Director of NASA’s Goddard Institute for Space Studies, sums up the problem as follows:

Model agreements (or spreads) are therefore not equivalent to probability statements. Since we cannot hope to span the full range of possible models (including all possible parameterizations) or to assess the uncertainty of physics about which we so far have no knowledge, hope that any ensemble range can ever be used as a surrogate for a full probability density function of future climate is futile.

And yet, this flawed approach is not only accepted, observance is actively encouraged amongst the climate modelling community. Again, Gavin Schmidt:

Yet demands from policy makers for scientific-looking probability distributions for regional climate changes are mounting, and while there are a number of ways to provide them, all, in my opinion, are equally unverifiable. Therefore, while it is seductive to attempt to corner our ignorance with the seeming certainty of 95-percent confidence intervals, the comfort it gives is likely to be an illusion.

One can, of course, speculate as to why such a flawed approach should have become an accepted best practice, and if you stay around, I will do so. In the meantime, what I want you to appreciate is this: confusion between variability and incertitude lies at the heart of the way in which climate model uncertainty is presented to policymakers. Parametric incertitude when modelled using Monte Carlo simulation represents a misapplication of an aleatoric method; furthermore, the confusion between incertitude and variability is compounded in the case of ensembles of differently structured models such as CMIP6 whenever their spread is treated as aleatoric uncertainty. These are issues that deserve to be better known by the policymakers who say they are following the science.

Probabilistic attributions for extreme weather events

To be, or not to be. That is the question.

Hamlet (Act 3, Scene 1)

Another area of climate science that has become highly influential in dictating policy is that of probabilistic Detection & Attribution, in which counterfactuals are modelled to assess the impact of anthropogenic climate change. Groups such as World Weather Attribution (WWA) are using such techniques to deliver rapid assessments of extreme weather events, stipulating the extent to which climate change had made the events more likely (the Fraction of Attributable Risk, or FAR). The leader of the group, Professor Friederike Otto, has made no secret of the fact that she does this to apply political and legal pressure upon the fossil fuel industry.

Unfortunately, the WWA protocol is heavily dependent upon Monte Carlo methods, and so once again aleatory methods designed to analyse variability are being used instead to analyse incertitude. Essentially, by performing Monte Carlo simulations on multi-model ensembles to generate event probabilities with and without anthropogenic climate change, the WWA is taking epistemic uncertainties (structural disagreements between different climate models, unrepresented physical processes, grid-resolution limitations, etc.) and feeding them into a frequentist framework designed purely for aleatory randomness. The output looks very objective, but by conflating two distinct types of uncertainty into a single probability density function, the method commits what philosophers would call a category error. It all ends up looking like aleatoric variability, and the underlying incertitude is buried.

The problem here is the illusion of scientific precision. If the uncertainty is epistemic (the model physics are fundamentally incomplete), generating millions of Monte Carlo iterations does not bring you closer to the truth. It merely creates highly precise, narrow confidence intervals around a potentially biased evaluation. In short, no amount of statistical sophistication can make you know what you do not know, only discovery can do that.

This is another limitation that deserves to be better known by the policymakers who say they are following the science. When a study quantifies just how much more likely a heatwave was made by global warming, the policymaker or a judge in a climate lawsuit will interpret the FAR as an aleatory risk factor—like the odds when rolling a die. They may not realize that the number is heavily dependent upon subjective choices such as the selection of models for inclusion in the ensemble, tuning, and arbitrarily drawn spatial boundaries that define the event.

Nor will the policy-maker necessarily be aware of the extent to which the Monte Carlo modelling of variability has been used to ‘rectify’ incertitude caused by gaps in data…

Data assimilation – or the invention of data

A dream itself is but a shadow.

Hamlet (Act 2, Scene 2)

Anybody who has looked into WWA studies will be struck by just how much they rely upon simulations to fill in temporal or spatial gaps within data. The simulated data is provided using reanalysis products such as ERA5 (produced by the European Centre for Medium-Range Weather Forecasts) or localized interpolated station networks. These data sets are generated using Monte Carlo-style sampling of estimated values.² But assuming an aleatory randomness to fill a gap in sparse data is conceptually unsound. The gap is an epistemic gap — it is an incertitude, not an outcome of variability. A few things should therefore be borne in mind.

Firstly, state is being conflated with process. The missing information relates to a state that is subject to conjecture. Variability is a process and it tells us what might be the case given our understanding of the physics. It may or may not reliably shed light on the unknown state. The data generated by Monte Carlo is therefore a plausible fiction – nothing more.

Filling gaps in climate data is an infamously difficult undertaking, since the assumptions of climate system ergodicity (when dealing with missing history) and spatial stationarity (when dealing with missing weather stations) are both highly problematic. Ergodicity, for example, assumes that statistical properties (like mean and variance) do not change over time, and that a long enough look at a well-recorded part of climate history tells you everything about the unrecorded parts. The climate, however, is non-linear, non-stationary, and chaotic. Using localized, modern variability to patch historical gaps assumes a stability in atmospheric behaviour that simply does not exist over multi-decadal timescales. Also, the spatial stationarity assumption implies that the spatial variance and correlation structure is the same in the areas lacking stations to those that have them. Extreme events, however, are by definition highly localised, non-stationary anomalies, and so the assumption of spatial stationarity is notoriously weak for attribution purposes. For these reasons, quite sophisticated techniques are required to turn the Monte Carlo fiction into anything approaching a realistic picture.³

To the above we must add the problem that a real historical record is highly correlated and prone to persistent, structured anomalies (like a multi-week blocking high). Gap-filling that fails to simulate such structures will exaggerate the likelihood of extreme weather since the attribution studies will be using an unrealistically uneventful baseline.

Notwithstanding the efforts made to overcome Monte Carlo’s limitations, the bottom line is that they are an attempt to salve a condition caused by using an approach that is philosophically unsound. In fact, by running its own stochastic Monte Carlo simulations fed by data (ERA5) that was already contrived by using a basically stochastic simulation, the WWA is doubling down on the misuse of Monte Carlo. They are using aleatory methods to represent ensemble incertitude but are also using a baseline dataset that was itself assimilated using aleatory assumptions when the real issue is one of incertitude. It’s Monte Carlo mishap all the way down! This is why sceptics argue that the precise probability figures published after extreme events can be mathematical illusions. The group is calculating the odds of an event occurring but doing so within a mathematician’s dreamworld, rather than determining reality on an actual planet. Worst still, it is a dreamworld in which incertitude is consistently mistreated as variability.

Economic uncertainties

Two thousand souls and twenty thousand ducats will not debate the question of this straw!

Hamlet (Act 4, Scene 4)

Then we come to determining the economic damage that can be expected from climate change. Once again, the uncertainties here are entirely down to incertitude, and yet again, the first tool the analysts reach for is the one designed purely for the analysis of variability. And I am not the first one to point this out.

Prominent economists, such as Robert S. Pindyck at MIT, argue that models used to simulate how climate change affects global economies frequently misuse Monte Carlo methods. The technique starts by selecting a particular economic scenario. Then a particular Integrated Assessment Model (IAM) is used to simulate how the range of possible climate change can translate into a range of possible economic damage for the chosen scenario. The problem is that the parameters that define how damage correlates to climate change (such as the exact economic damage from a 3°C rise or the precise climate sensitivity) are unknown. To fix this, the economists attach arbitrary probability distributions to these unknown parameters, run thousands of Monte Carlo simulations, and take the average. Pindyck points out that this achieves nothing and merely creates the illusion of precision. If the underlying probability distributions are just guesses, the aggregated Monte Carlo output is also just a guess, dressed up to look like rigorous statistical analysis. Add to that, if one combines a different IAM with a particular economic scenario one will obtain radically different results — such is the parametric and structural uncertainties in these models. The epistemic uncertainties dwarf anything that variability brings to the table, and yet variability is the only paradigm used in the uncertainty analysis.

By now we have three layers of misuse of Monte Carlo. Firstly, the incertitude resulting from sparse data is rectified by using Monte Carlo perturbed physics simulations that are premised upon a presupposed statistical variability. These questionable data sets are then used as a baseline for climate risk calculations that are themselves based upon questionable Monte Carlo simulations that treat the subjective incertitude of the model ensembles as if it were objective variability. Then, finally, probability distributions for economic parameters are invented simply to enable a further layer of questionable Monte Carlo simulation that translates climate change uncertainty into incertitude relating to economic damage. At every stage, a technique designed to analyse variability is being mis-used to analyse incertitude, and at no stage is anyone saying, ‘Wait, this is not how the real world works, we are just playing with fancy statistics to give the whole thing a look of scientific rigour’.

Fat tail fantasies

Each toy seems prologue to some great amiss.

Queen Gertrude (Act 4, Scene 5)

Whilst we are talking about the Monte Carlo simulation of economic damage, we need to address the elephant in the room – or perhaps the elephant that was never there. I speak, of course, of RCP8.5.

When researchers sample the extreme upper-bound tails of an ensemble that is paired with an extreme scenario such as RCP8.5, they are compounding extremes. It effectively means modelling the worst-case possibilities of a world that is already a worst-case scenario. When a stress test frames the extreme tail of an already extreme scenario as a benchmark for risk, it ceases to function as a plausible stress test and morphs into a work of science fiction.

The problem, of course, is that at no stage has this particular stress test been recognised as the fiction it is.⁴ Instead, it has been presented to policy-makers as a very scientific-looking ‘business as usual’ benchmark. But as Schmidt pointed out in relation to ensemble uncertainties, the push is for analyses that are scientific-looking, in which the reality doesn’t seem to matter as much as the impression formed. But the reality does matter when that reality is an approach that violates how risk-management frameworks are supposed to work. Of all the issues that deserve the better understanding of policy-makers, this is perhaps the most significant since it has led to several major consequences. It is essentially the basis for most of the existential fear-mongering that seems to play such a central and undeserved role in the climate change debate.

Summary – or what on earth is going on here?

Our indiscretion sometimes serves us well.

Hamlet (Act 5, Scene 2)

In conclusion, by extensively using Monte Carlo simulation, climate scientists are taking a methodology designed to analyse variability but using it to analyse incertitude. And the fundamental problem is this: if the uncertainty is due to incertitude, the probability distribution being sampled is not really a true probability distribution. It is a curve with a shape that is supposed to model our epistemic ignorance, but it would be fairer to say it is a pure invention plucked out of the air to enable the computation to take place. By treating an invented curve as a true probability distribution, the computation produces a highly precise numerical range. But this is a mathematical trickery that transforms a state of ignorance into a state of objectively held confidence, thereby creating a dangerous illusion of control for policymakers.

I have outlined how this flaw can be found when dealing with economic damage (IAMs), physical models (CMIP6), and gaps in observational data (as often encountered in extreme event attribution). That pretty much covers all of the science that policymakers are so keen to follow, so it is important that scientists should be getting this right. Which raises an obvious question. If getting it right is so important, why are the climate scientists seemingly so relaxed about using a technique that is getting it wrong. In a word, the answer is pragmatism.

In trying to quantify uncertainty, climate scientists believe themselves to be confronted with a stark choice: either accept an invalid statistical compromise, or abandon the study entirely. But there is no way that their studies could be abandoned. So, they use Monte Carlo despite it being philosophically invalid. Essentially, they see it as the only computational tool capable of providing a continuous dataset that a computer can actually run. As Freddy Mercury once said, the show must go on.

Despite the harshness of my criticism, I have to admit that I have a certain amount of sympathy for the climate scientists’ position. And I would be the first to accept that many scientists recognise the issue and make considerable effort to mitigate the risks of miscalculation.⁵ If anything, the real message of my article isn’t so much the incorrect calculation of risk that can result from misuse of Monte Carlo, it is the resulting false confidence. The accepted approach takes subjectivity and incertitude and cloaks them in apparent objectivity and scientific rigour. Modelled ignorance is masquerading as an objective calculus, and it is this false impression that is doing the damage. Policymakers see the sciencey numbers and the attendant confidence intervals without realising some of the shaky philosophical foundations upon which the calculations are made.

The policymakers need to be better informed, and one might expect the scientists themselves to do more to help by drawing attention to the dubiety of their methods. However, one suspects they don’t want to do so because it gives ammunition to the sceptics. So, they indulge in the noble lie, leaving policymakers in their naivety, whilst simultaneously failing to realise that such a lack of candour is the very thing that fuels the scepticism. At the end of the day, there is no substitute for scientific integrity.

As a final comment, I should add that I would have more sympathy for the climate scientists if they would make a greater effort to seek alternatives to the probabilistic charade that Monte Carlo offers them. Such alternatives do exist, but there seems limited appetite to explore them. It is true that critics of probabilistic Detection & Attribution have offered their own storylines approach as a way of escaping the Monte Carlo peril⁶, but this has received pushback. There are also non-probabilistic methods available to the climate scientists, such as Dempster-Shafer theory, fuzzy logic, possibility theory and Info Gap decision theory, but their application has been limited. Perhaps climate science statisticians are too invested in the old ways and occupy a position of trust in the eyes of policymakers that has led to complacency. Besides which, the policymakers want to see those probability distributions and confidence intervals. Consequently, the climate scientists probably see no reason to change their approach. In the meantime, the show does indeed go on. And on, and on…

Footnotes:

[1] If one takes the philosophical position that all real-world variability is ultimately deterministic, then the two causes of uncertainty collapses to one, i.e. incertitude. In practice, even when deterministic, the world can still exhibit stochastic behaviour, leading to an uncertainty that is epistemically irreducible for all practical purposes, so I will stand by the view that there are two causes.

[2] Kriging techniques generate the mean and variance values and Sequential Gaussian Simulation (a form of MC) is then conducted to generate what is meant to be a plausible and realistic data landscape.

[3] I do not intend going into detail in this article. It is just far too big a subject.

[4] Well, I say that, but sceptics have for a long time tried to tell anyone who would listen. RCP8.5 has finally been retired, albeit without the admission that it was never plausible in the first place. One also suspects that it will continue to enjoy a zombie influence in many future studies.

[5] For example, the WWA rejects the output of a Monte Carlo analysis if it suggests a finding that lacks a plausible physical explanation or contradicts historical observations. I’ve also skipped over a lot of detail regarding how missing data is simulated – detail that would include sophisticated attempts to make the data simulation as realistic as possible.

[6] The storylines approach is not without its problems, as gleefully pointed out by the proponents of probabilistic Detection & Attribution. I wonder if the policymakers are aware of the extent to which there is such a fundamental difference of opinion lurking within the so-called consensus of ‘settled science’. These are not two camps offering complementary methods; each believes the other’s approach to be invalid.