Uncategorized

When More is Less

A recent study by Jonathan Bamber et al solicited expert opinion regarding the possibilities for Sea Level Rise (SLR) resulting from Antarctic ice sheet collapse. The study’s methodology included a technique known as ‘Structured Expert Judgement’ to improve the reliability of the conclusions drawn by aggregating the opinions within the expert community. It has concluded that the IPCC AR5 estimates are now understood to be too conservative. According to the paper:

This study suggests that experts’ judgments of uncertainties in projections of the ice sheet contribution to SLR have grown during the last 6 y and since publication of the AR5. This is likely a consequence of a focused effort by the glaciological community to refine process understanding and improve process representation in numerical ice sheet models.”

The study is quick to defend the apparent paradox that greater understanding can lead to greater uncertainty:

This negative learning…may appear a counter intuitive conclusion, but is not an uncommon outcome: as understanding of the complexity of a problem improves, so can uncertainty bounds grow.”

But how, I hear you ask, can a greater level of uncertainty be seen as an improvement? Once again, the paper wastes no time in providing the explanation:

We note that for risk management applications, consideration of the upper tail behavior of our SLR estimates is crucial for robust decision making. Limiting attention to the likely range, as was the case in the Intergovernmental Panel on Climate Change AR5, may be misleading and will likely lead to a poor evaluation of the true risks.”

This is not the first time that I have come across climate scientists asserting that “consideration of the upper tail behavior” leads to a determination of “true risks”. Nor is it the first time that I have encountered the perils of allowing “refined process understanding” to create an increased perception of risk.

I worked for some time in the field of safety risk assessment – long enough for an important principle to become apparent: The more sophisticated and detailed the risk assessment, then the greater will be the estimated risk level. This is as a direct result of the increase in the number of extreme impact possibilities that are identified by the ‘enhanced’ analysis, combined with the greater uncertainty associated with the low probabilities that should be attributed to them. In transport safety analysis (my particular field) this effect was captured by the following apocryphal set of safety assessments, each more sophisticated than the previous:

Basic analysis:

Q: What is the worst case scenario for a failure of this particular road safety system?

A: A road traffic accident resulting in fatalities.

More sophisticated analysis:

Q: What is the worst case scenario regarding road accident fatalities

A: One that involves a busload of schoolchildren

Even more sophisticated analysis:

Q: What is the worst case scenario regarding the death of schoolchildren

A: That their bus collides with a nuclear fuels convoy

This sequence of analyses was known in the trade as the ‘school bus – nuclear convoy scenario’. The point was this: If one analysed a road safety system long enough, one would always end up with the ‘school bus – nuclear convoy’ scenario, and that is no good when trying to decide the required safety integrity of a given safety-related system. Yes, one can design all systems to be catastrophe-proof, irrespective of likelihood, but that’s not how safety management works. Terms such as ‘reasonably practicable’ apply, and so safety assessments are constrained to only address primary outcomes (e.g. road fatality) without speculating upon the full range of possible, non-zero-probability contexts (e.g. school buses and nuclear convoys). Put another way, if you allow yourself to engage in conjecture within areas of high uncertainty you will simply become a hostage to your imagination.

So when climate scientists declare that a further 6 years of modelling of ice sheet dynamics has led to confident proclamations of an increased 90th percentile SLR impact (courtesy of increased model uncertainties) I have to ask whether they are simply succumbing to the thrall of a ‘school bus – nuclear convoy’ scenario. Their models may be more sophisticated, but if that sophistication is simply adding to the uncertainty (glibly attributed to increased complexity) then they are inferior models for the purpose of risk assessment. Sometimes it is important to refrain from adding complexity and concentrate instead upon the evaluation of a more tractable, albeit simpler risk model.

Of course, this wouldn’t be an issue if it were not for the climate scientists’ habit of defining risk in terms of the worst-case scenario impacts that current levels of uncertainty fail to discount. Such an approach simply allows the increased uncertainty to open the door for more speculation of the ‘school bus – nuclear convoy’ variety. The scientists continue to talk of risk aversion and the identification of the “true risks”, but the reality is that they are just replacing risk aversion with uncertainty aversion. No amount of structured expert judgement can save you from this problem.

In conclusion, Jonathan Bamber et al may believe that they have discovered something new about the reality of the “true risk”, but they should not be so hasty. Even if they can argue that previous levels of uncertainty are no longer scientifically justified, they cannot argue that increased levels of uncertainty provide a better basis for risk assessment. As far as I am concerned, the experts consulted could just be falling into the trap of basing their ‘enhanced’ assessments on the climatological equivalent of ‘school bus – nuclear convoy’ speculations. Give them another 6 years and their uncertainty will flood the whole world.

6 thoughts on “When More is Less

  1. Thanks for your support Richard. It has occurred to me since posting this article that there may be a more explicit way of summarising the point I am trying to make:

    What Jonathan Bamber and his colleagues are saying is this:

    “The previous assessment of risk was based upon an unrealistically low level of uncertainty. Now that the uncertainty has been increased, we have a better understanding of the risks.”

    What they should be saying is this:

    “The previous assessment of risk was based upon an unrealistically low level of uncertainty. Now that the uncertainty has been increased, we should no longer be claiming to understand the risks.”

    But it doesn’t end there. One has to properly understand why the uncertainty has increased. If it is due to increased complexity that has actually reduced the predictive skill of the model, can we say that the new levels of uncertainty are actually more realistic? Or are we just introducing gratuitous uncertainty by entering areas of increased speculation?

    Liked by 2 people

  2. I have vague memories of ‘maximum credible accident’ being used in the nuclear power industry when I worked there in the early 1970s. My own work did not involve such cogitations, but I remember that, for example, for one operating reactor the maximum credible accident was a light aircraft crashing into coolant pipes – there being a small airfield somewhere in the vicinity. That I suppose provided some defence against an emotive slide into a ‘school bus nuclear convoy’ approach. It was presumably understood that one could always come up with something sufficiently dire against which there was little the designers and operators could do to cope.

    Like

  3. Personally, what I fail to understand is the focus on the upper tail end behaviour when assessing ‘true risks’. The ‘true risk’ is surely a function of the entire distribution? The worst case scenario, made even worse by ‘better knowing how much we don’t know’ may become even more scary and extreme, but why does the upper tail end extend even further into the hinterlands of a future climate apocalypse, while apparently the lower tail end does not expand equally along the opposite direction into ‘No Worries Land’? It seems to me that the generation of increasingly dramatic worst case scenarios is only meaningful if the whole distribution moves to the right, maintaining its shape, whilst the most likely range becomes a little more worrying as it shifts into riskier territory. But what these people seem to be doing is just stretching the upper tail end into ever more extreme territory with ever decreasing likelihood as accompaniment, claiming that this is somehow a valid assessment of the ‘true risk’. The ‘most likely’ range does not move however.

    Liked by 2 people

  4. Didn’t the fat upper tail problem arise from speculation among some frogs (or more likely newts) relaxing in the sauna? “Nice and steamy in here” said one. “Yes, but what if…?” interjected another. And so they all hopped out and went back to leading a sustainable life in the muddy bottoms.

    JAIME
    The lower tail is constrained at zero, so when you increase uncertainty and therefore depress the modal (most likely) tip of your graph, the probability has nowhere else to go but towards the upper extreme, a bit like the air in a sausage balloon. This is Lewandowsky’s Paradox, and it only works with a distribution highly skewed towards the left (zero) because otherwise there’s no reason why increased uncertainty should make the worse more likely. This is necessarily the case in risk assessment, given that the point of the exercise is to reduce risk, so there’s an unspoken assumption that the purpose is to get the modal peak as close to zero as possible. The argument no longer works in a situation where you’re willing to accept high risk, e.g. a war, where you’re willing to send a platoon into combat even though there’s a high chance they’ll all get killed. Fog? No information on enemy strength? So what? Over the top. Which is what climate statistics is.

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.