‘Twas the night before Christmas, when all through the house, not a creature was stirring, not even a mouse…

Well, when I say ‘not a creature’, I mean to exclude those that were stirring over governmental impotence in the face of an international crisis. I think I may also be ignoring those who were stirred by tales of stolen elections, jackboots crushing civil liberties and transgender madness. In fact, come to think of it, ‘twas the night before Christmas and all hell was breaking loose. They say Christmas is a time for all the family to get together – before tearing a lump out off each other. Well, if the Cliscep family is anything to go by, this tradition appears to be going from strength to strength. A bit more stirring is the last thing that Cliscep needs.

For that reason, I now wish to turn to a subject that is so bereft of emotional valence, so soporific to discuss, and of so little interest to the vast majority of you, that there can be no risk of disharmony breaking out. Yes, I’m afraid this is going to be like one of those New Year’s Day lectures you used to sit down to watch on television once the festive fisticuffs had petered out. But instead of an audience of middle-class, privileged, swotty little kids, I choose instead for my audience a group predominantly featuring the disenchanted white elderly male. All I ask is that you press the like button as though I had made my point clearly and as if you appreciated the angst-free distraction it provided.

My Chosen Subject

For this year’s lecture, I choose to cover the subject that I believe to be at the centre of the climate debate and one that has been my main preoccupation on this website for the last three years: Uncertainty.

Specifically, it is my intention to answer a question that was asked by Thomas Fuller over on the ATTP website shortly before Christmas:

Just how do you go about quantifying subjective uncertainty?

Given the professed expertise shared by the denizens of ATTP, one might have expected an orderly queue of volunteers lining up to answer the question. After all, it is a perfectly reasonable question to ask and it does have a perfectly reasonable answer. But, alas, it seems they all had better things to do. Never mind. I’m sure what I have to say below was on the tips of their tongues.

Defining terms

Before I start, I should point out that the term ‘uncertainty’ means a lot of different things to different people, and this applies even more so to the expression ‘subjective uncertainty’. This makes any differences of opinion on this subject difficult to analyse. Consequently, it would be wise of me to begin by clarifying what I mean by uncertainty.

Put simply, uncertainty is the condition that appertains when the probability ascribed to the truth of a statement is neither unity or zero. Such uncertainty may be due to either an inherent variability relating to the factors that determine the probability, or an incertitude arising from lack of information. The former, I shall refer to as objective uncertainty. The latter, I shall refer to as subjective uncertainty. I shall also be making the assumption that Thomas had this distinction in mind when he asked his question.

In practice, it is rarely the case that uncertainty falls neatly into one or other of the above classes. This is a problem because the propagation and management of uncertainty differs according to class, and a number of misconceptions can arise if the two basic sources of uncertainty are not carefully separated. But, for today, let us stick to the question of quantification.

The Best that Probability Can Do

The first clue regarding the correct quantification of subjective uncertainty lies in the fact that probability is its hallmark. For objective uncertainty, probability distributions may be constructed that reflect the likelihood that a variable of undetermined value may lie within a specified range when measured, and it is the objectively determined vagaries of nature that lie behind the shape of that distribution. For subjective uncertainty there may be little objective basis upon which to construct such a distribution, so this leads many to assume that the probabilities cannot be quantified. But this is not the case. Probability has always been an ambiguous concept that applies equally to both objective and subjective uncertainties, and can therefore be used to quantify both classes. The real question is how to interpret the quantification and, in particular, how one can judge the proximity of the subjective quantification to any objective reality.

For example, a fair coin will land on heads with a probability p=0.5. But if I were to tell you only that you are dealing with a two-state system that has adopted one of its two states, what probability would you ascribe? In the absence of any further information you should still ascribe p=0.5, since this is the expression of maximum epistemic uncertainty relating to the nature of the two states. Upon learning that you were dealing with the flip of a coin, you would update your calculation but it would, in this example, still remain at p=0.5. All subjectivity and ambiguity has been removed, but the overall uncertainty appears to be unchanged. In this example, the subjective probability and objective probability happen to be the same.

Given that I defined uncertainty in terms of deviation from p=0 and p=1, it should come as no surprise that p=0.5 represents maximum uncertainty for a two state system. In an n-state system the maximum subjective uncertainty is represented by p=1/n for all states. Of course, as further information is revealed, some states may become more favoured and the probability distribution will become less flat. In the general case, the uncertainty (H) encapsulated by a probability distribution is given by Shannon’s equation for entropy:

H = – Σ p * loge(p)

It makes sense that this entropy reflects uncertainty, since the less information is contained in the signal, the less confidence one can have in what it is telling you.

Shannon’s entropy formula applies irrespective of the degree of subjectivity and so it provides the answer to Thomas’s question. However, it should be immediately apparent that the question itself ducks the issue. The real question isn’t the quantification of subjective uncertainty, or indeed the quantification of the extent to which the uncertainty is subjective. The real question is how far, in a given instance, is the value given by Shannon’s equation from the value one would get if the uncertainty were entirely objective. The problem with this question is that it cannot be answered in advance of the subjectivity being removed. However, a measure of confidence can still be quantified if one abandons probability theory and concentrates instead upon the strength and heterogeneity of evidence available and what this tells you regarding justified belief. For this you need a non-additive mathematical approach such as that provided by possibility theory.

Beyond Probability

The confidence of which I speak is the confidence to be gained when there is a big difference between the possibility of something being the case and something not being the case. This is analogous to the low uncertainty associated with the probability being close to either zero or unity. However, in possibility theory, the weight of evidence supporting each possibility is used as the basis for determining the calculation. I have covered this subject before on Cliscep, and so I will not repeat myself here. Those wanting to know how such a quantification is performed may follow this link and read what is said under ‘Confidence Measured Objectively’. Suffice it to say, the calculation accounts for the weighting of evidence and reflects what this weighting has to say regarding both the possibilities and what is already necessarily the case. The relevant formula is:

Confidence(x is A) = Possibility(x is A) + Necessity(x is A) -1

This is a confidence that is cognisant of what the evidence is telling you but also takes into account the credence one should place in the evidence. As such, it delineates and accommodates the concepts of likelihood and ignorance in a way that probability cannot. More to the point, it is a quantification of subjective uncertainty that captures the extent to which the objective reality remains hidden.

A Reflective Moment

For those that have made it this far, I commend you for your appreciation of the undramatic yet mysterious world of uncertainty analysis. But I am under no illusions and suspect that this article will attract little interest. Who wants Open University when the real-life soap storylines are so dramatic nowadays? This was not an article to get the heart pumping. There are no villains to boo, no heroes to cheer. The only discordance on show is the probabilistic discordance measured by calculations of entropy. And yet I hope there was still enough to engage the reader. The knowledge that uncertainty (and even the extent to which subjectivity is an issue) is amenable to mathematical calculation should be a source of wonder and comfort to those who seek reassurance that the world is not completely intractable. These mathematical insights are out there, waiting for those who are inquisitive enough to investigate. They are vital for anyone wishing to quantify the uncertainties that bedevil the interface between science and politics. And there was simply no excuse for those who chose to ignore Thomas’s question.

29 Comments

  3. John, Got as far as definition of subjective uncertainty, at which point certainty became my objective and I dozed off. Then read the last paragraphs and pondered what it had to do with Christmas fisticuffs that had first intrigued me. I shall live in profound ignorance, other than Cliscep beats ATTP again. Hurrahs.

    Liked by 1 person

  4. John great post, in that it makes the mysterious seem graspable, even if I haven’t. And you think a special mindset is required to grasp evolutionary matters 0:

    Unless my innate glaze of such matters, through which I forced myself to read the post, is too limiting, don’t you have ‘former’ and ‘latter’ the wrong way around in the middle paragraph of ‘Defining Terms’? If there is variability relating to determining probability factors, isn’t that the Subjective breed?

    “The knowledge that uncertainty (and even the extent to which subjectivity is an issue) is amenable to an objective mathematical calculation should be a source of wonder and comfort to those who seek reassurance that the world is not completely intractable.”

    I am standing beside myself with wonder. However, regarding practical application, isn’t rounding up the possibilities and assigning weights to them essentially a matter of expert opinion? So notwithstanding we have many ways to minimize bias in same (which are oft subverted), isn’t this ultimately subjective?

    Like

  5. Andy,

    Thank you for the feedback. Something has just come up so I can’t do your comment full justice right now. I’ll get back to you. In the meantime, ‘inherent variability’ is thought to be objective uncertainty because it is inherent to the system being analysed and is therefore a feature of the real world. The incertitude, on the other hand, is subjective since different observers will experience different levels of uncertainty depending upon the information that is available to them. Hope this helps.

    Alan,

    My apologies for the effect it had on you. I fear you may not be the only one though.

    Liked by 2 people

  6. Well, I can’t profess to understand much beyond the first few paragraphs of John’s post here but I’ve come up with a handy application of Shannon’s Entropy. We could say that on Trump and Covid posts, Shannon’s Entropy is near 1 when applied to the probability of a comment being in either one ‘camp’ or the other (especially Trump posts). At ATTP, the probability of an informed answer to Thomas’s question along the lines of what has been written here has turned out to be zero, so I guess that would mean that applying Shannon’s Entropy to TF’s subjective uncertainty query at ATTP – in terms of the probability distribution of intelligent, informed responses – gives the result H=0. This is not to say that a similar situation might exist here at Cliscep in terms of meaningful, informed responses (or not) to John’s very technical post!

    Like

  7. John, “In the meantime, ‘inherent variability’ is thought to be objective uncertainty because it is inherent to the system being analysed and is therefore a feature of the real world. ”

    Ah… got it! Thanks.

    Liked by 1 person

  8. John, I see you used the image that PZ Myers uses as background when he’s quoting someone he disagrees with on his blog, Pharyngula. I’ve often wondered where it came from –. I’m guessing from some Monte Python animation?

    Like

  9. Andy,

    “So…isn’t this ultimately subjective?”

    You are quite right. I was trying to make the perhaps overly nuanced and unnecessary point that the quantification of the subjectivity is objective in the sense that once the areas of subjectivity have been agreed upon, there are calculations that anyone can then apply that will give everyone the same answer. I may consider clarifying the article.

    Mike,

    It’s a Monty Python character called Eric Gumby. Google “My brain hurts Gumby”.

    Liked by 2 people

  10. John,
    The context of Tom’s question was this post which was about the sea level projections presented in the AR5 report. Tom’s objection was to the following comment

    This is particular challenging to model, and IPCC write that a collapse may cause sea level to rise faster than the ‘likely range’, but also that this risk is essentially impossible to model (paraphrased).

    This is then followed by

    In order to quantify the risk we therefore had to look to other lines of evidence, and here we looked at an expert elicitation which quantified the subjective uncertainty within the community of ice-sheet experts.

    which seems broadly consistent with my response to Tom. Since that seems, according to you, to be wrong, what would you do instead?

    Like

  12. Silly Beth you’ve posted that da Vinci image the wrong way around so the writings backwards and it’s more difficult to read. Not sure the water is swirling correctly either. Don’t think Leonardo ever travelled to your turnip emporium to see water flowing widdershins.

    Like

  13. ATTP,

    In the context of a discussion regarding the impact of sea level rise, on Dec 19, 8.05pm, Thomas asked the following:

    “I mean–quantifying subjective uncertainty? How does one do that?”

    To which, on Dec 20, 8:13am, you replied:

    “I don’t know, do you just assume there isn’t any?”

    Now I know you were not playing ‘Who Wants to be a Millionaire’, but I note that was your final answer. What you are posting here is a statement that claims that expert opinion was elicited to quantify the subjective uncertainty. And yet you think this is broadly consistent with what you told Thomas?

    Like

  14. ATTP,

    To be fair, I should point out that you did elaborated later by saying:

    “No, I don’t think it’s simply expanding error bars. It’s, as I understand it, using other lines of evidence to try and determine by how much they should be expanded.”

    Like

  16. ATTP,

    The problem is, you had me at ‘I don’t know’.

    Its all very well claiming now that you had already answered the question by sending a link, but clearly it is the methodological approach followed by the glaceologists that prompted Thomas to ask his generic question challenging the concept of quantified subjectivity. This question required a generic answer, and I do not see on ATTP anyone, including yourself, providing that answer.

    Like

  18. John,

    The problem is, you had me at ‘I don’t know’.

    Yes, I realise this. I wasn’t expecting anything different. The point, though, is that I don’t really know the answer. Having already provided a link to a set of posts that explained what had been done, I didn’t really see myself as being in a position to try and explain it any differently.

    Liked by 1 person

  19. ATTP,

    That’s a fair and honest answer. But what I would add is that it is a shame that only David Benson seemed willing to help Thomas with the essentially philosophical nature of his question. It is even more regrettable that his efforts were dismissed as ‘hand waving’, even though at least one of his links was to an article that was highly relevant. I do believe there are some fundamental issues that your website is reluctant to address, and those that try are usually met with resistance. I hope that you appreciate that I am trying to address some of those issues here at Cliscep, even though you may not agree upon their importance.

    Like

  20. Beth ‘‘twas quite proud of my nested Leonardo witticisms but then no one seems to have identified them. da Vinci wrote using mirror writing so, if I pretended not to know this, I could accuse you of sending the swirling water image and accompanying writing back to front. But if you had done this, then any clockwise motion of the water would have been converted into an anti-clockwise motion (= widdershins). One of the first things I did upon arriving in Australia during a visit was to run a tap and watch the water flow widdershins as predicted. But daVinci never travelled to the Southern Hemisphere.

    It’s true what they say, never explain your jokes.

    Liked by 1 person

  21. I knew he wrote backwards, Alan.,Re backwards water, I remember testing this when I first visited London town as a young serf and turned on the tap in the bathroom. Whole new experience. 🙂

    Like

  22. I enjoy these discussions of risk and uncertainty that you post, John, in fact I’m still ruminating from time to time on the gambling one you did long ago. I have seen the confidence equations before, but this time it has occurred to me that putting a figure other than 0 or 1 on a concept like necessity (which seems to me an absolute) is not readily understandable. I wonder if you could come up with an example from real life to help me understand how necessity can be other than 0 or 1.

    Like

  23. Geoff C,

    Thank you for the feedback.

    I find this a difficult subject to explain, so I will attempt a hand-waving answer but also provide you with a link to a paper that explains it far better than I could. It also uses diagrams and worked examples to make the point. First, the hand waving:

    In the words of its inventors:

    “Possibility theory is an uncertainty theory devoted to the handling of incomplete information. As such, it complements probability theory. It differs from the latter by the use of a pair of dual set-functions (possibility and necessity measures) instead of only one. This feature makes it easier to capture partial ignorance.”

    The point to take away from this is that neither ‘possibility’ or ‘necessity’ are probability, but they combine in a way that gives a form of imprecise probability. In fact, possibility theory is what is known as an evidence theory, and the ‘possibility’ and ‘necessity’ are complementary concepts within the theory that measure the strength of evidence. In the absence of evidence, all things are possible and equally possible. As evidence accumulates, however, the justification for believing one possibility over the alternatives may increase – it becomes more possible to believe whilst the alternatives become less possible. This variance is captured by a possibility distribution (not to be confused with a probability distribution). However, absence of evidence is not evidence of absence. So to complete the picture one has to consider the degree to which a given possibility remains allowed by all the evidence, i.e. not just the extent to which a possibility is not ruled out but the degree to which it is ruled in. This is where necessity comes in. Put formally, the possibility degree evaluates to what extent event A is consistent with the knowledge π , while the necessity degree evaluates to what extent A is certainly implied by the knowledge.

    Now for the proper explanation:

    http://home.iitk.ac.in/~partha/possibility

    Like

  24. @ John

    Re: the Shannon index. This is easily recognisable to an ecologist. In ecology it’s called the Shannon diversity index, & it is complementary to the more used Simpson’s diversity index. Interesting that a metric for diversity is also used for entropy. In the world of biological communities the most diverse case is where each species represents the same proportion of the community. This presumably can be seen in entropy as everything being evenly spread out. Your uncertainty is my diversity. Sort of the same thing.

    I mention this because I had no idea that the Shannon index had a life outside ecology, & indeed had no idea that it might have originated elsewhere and been ported over to squishy science. I suppose now I had better look up its origin.

    Like

  25. Wow, and as a software person I’m amazed that the Shannon index ever made it to ecology! Actually that’s not quite true. But Shannon was so key to information theory.

    Like

  26. Jit,

    And until recently I had no idea that Shannon’s entropy had an application in ecology. It seems to crop up all over the place when you look for it.

    Incidentally, I am pleased to report that your puffin article has now received over 250 views (more than my latest post). As a comment on my Birdaggedon article it was destined to receive about a dozen views. If you have any further such ‘comments’ from which articles can be created, then I’m sure Cliscep fans would be interested.

    Liked by 1 person

  27. Thanks John, I’ve had a quick look. It looks as though nec being other than 0 or 1 only occurs with fuzzy sets, but I’ll read it again tomorrow. It all looks a bit complicated, I may have to look into this field and reorganise it…….

    Like

  29. @ John there are a few things at the nexus of climate change and ecology that I was only able to touch on in my defiant yet futile blow against climate alarmism (my little book of climate scepticism includes a fuller account of the puffin story). So I can certainly come up with something in due course, although whether it will meet the required standard is another matter.

    Speaking of which, I finally have a few copies of Denierland here so I can send one to anyone who’s interested. Lemme know. One can also be had from Mr Bezos for nine quid.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.