A lot has been said in the news recently regarding the safety of AI systems. In particular, there have been testimonies given by people involved in AI development that warn of existential risks materialising within the next decade. Understandably, this has got a lot of people hot and bothered, either because they take such warnings seriously, or because they believe the whole thing to be outrageous scare-mongering. Some of those in the latter camp have suggested that the hype is the result of a cynical attempt to promote the capabilities of a particular brand of AI, or even an attempt to curb the development of competitors in order to either maintain a leadership or to reverse one. Such controversies are made possible because we are dealing with future uncertainties, and when the uncertainty of the future is involved, all manner of rhetoric can be let loose.

However, with my software safety engineer’s hat on, I have to say that much of the public debate seems to miss the point. This shouldn’t be a debate about speculative superintelligences or armies of evil robots; it should be centred on the certainties of the present. In particular, there is one present-day fact that settles the debate decisively. As it stands, there is a vast range of potential AI applications – specifically those involving a System of Systems (SoS) with Artificial General Intelligence (AGI) components – for which software safety engineers currently would not have the capability to create a reliable, watertight safety case. I am not saying the production of such a safety case would be difficult; I am saying that, unless someone comes up with new safety case technologies, the problem is intractable. So, there is a very easy explanation for why insiders are concerned; it is because they know that for a great many proposed AI applications, it would be impossible to prove the safety of the system, either in advance of its deployment or in real-time following its deployment. And yet this gaping void in safety assurance capability is not putting a brake on the development of AI, or indeed on proposals for AI applications. In the absence of a brake that enables safety case technologies to catch up, we are destined to be taking unquantifiable safety risks that may very well prove to be catastrophic, dependent entirely upon the extent to which control is incautiously ceded to AI.

The situation before AI

To explain why this is the case, I first need to explain some basic terminology and clarify why, even without AI being involved, the challenges of proving system safety can nevertheless be substantial. Firstly, I need to explain the concept of the safety case.

Traditionally, a safety case has taken the form of a dossier of documents that can be submitted to a Functional Safety Assessor in order to evaluate whether or not a system will be acceptability safe in operation. For larger and more complex systems, such dossiers can run to thousands of pages, comprising, but not limited to: evidence of the integrity of the development process, the competence of all involved, the handling of any Software of Unknown Provenance (SOUP), supply chain management, the results of requirements validation, design review results, static and dynamic test results, and operational compliance with specific safety requirements. In essence, it is the proof that the quality assurance regime has met the standards required for the level of safety integrity demanded of the system (usually expressed in terms of a safety integrity level such as an EC 61508 SIL).  Whilst absolute safety cannot usually be assured, demands for sufficiency of safety become increasingly stringent, in accordance with the safety criticality of the system, so much so that the very highest SILs demand a vanishingly small risk of an unsafe failure. This is where the System of Systems (SoS) becomes a problem.

A fundamental prerequisite of a safety case is that the full space of states that the system may enter during operation should be considered and accounted for. Anyone involved in software development will know how difficult this can be in practice, but when dealing with a SoS (a modular system that comprises systems in their own right) the problem is greatly compounded because the system state space is vastly greater and much more difficult to predetermine. This is primarily as a result of emergent behaviour, whereby risks often arise from the interaction of multiple systems rather than from any single component failure. This means that the safety case of the SoS cannot be compiled simply by collating the safety cases of its component systems. This problem is often exacerbated in practice by the fact that the component systems typically have individual development histories controlled by differing organisations, employing different standards. Distributed ownership can also render traditional testing across the entire combined network prohibitively expensive or logistically impossible. In short, an SoS combines the most demanding of assurance challenges with the most compromising of circumstances. Even before AI gets involved, safety engineers have been struggling to meet the challenges of producing a foolproof safety case for a System of Systems.

One of the ways in which the safety engineering community is looking to meet this challenge is through the use of the Dynamic Safey Case (DSC). A DSC for a System of Systems shifts the safety argument from a static, paper-based document signed off at design-time to a live, software-driven process that executes during runtime. Instead of asserting in advance that the system will be sufficiently safe under all possible circumstances, the DSC continuously assesses and generates evidence of system safety in real-time. To do this the DSC not only confirms that the system is currently operating safely within its known boundaries, but that it can be mathematically proven that the current risk level remains acceptable – at least as it stands. As such, a DSC is essentially a software-based safety case* that is continuously matching real-time evidence against design-time assumptions. Through such real-time assessment the DSC can address the dynamic and unpredictable nature of an SoS, dynamically altering the thresholds of the safety controls to allow for changes in the environment and the system’s relationship with it. Furthermore, the DCS can intervene in system operation whenever the emergent boundary violations are detected. However, it should be noted that not all applications lend themselves to feasible, run-time monitoring or to real-time intervention. This is true of a conventional SoS but it becomes especially true once AI enters the scene.

Now add the AI

Whenever experts want to draw attention to the controllable risks posed by AI, they tend to illustrate using systems that lend themselves to physical safety controls. This usually entails entirely separate and independent physical safety hardware that the AI cannot interfere with. Essentially, whilst the AI may develop the desire to follow a path leading to human ruin, we can physically wire the AI system in such a way as to render that path impossible. The system is thereby bounded in such a way that the AI would have to break the laws of physics to get its evil way.

Such applications do exist and I think it is fair to say that the safety engineering community is on top of them. In such instances, the safety case is relatively easy to construct since it is built on the proof that the AI lacks the physical pathways to act upon any errant ‘thoughts’.

However, for a large percentage of real-world AI applications the possibility of physical containment is an illusion. For example, physical barriers are not germane for applications in which AI is managing global corporate logistics, cloud infrastructure, or financial trading systems. Nor is it an option when it is acting as a strategic corporate advisor, a legal framework optimizer, or a national security intelligence analyst. And don’t even think of physical barriers bailing out the safety engineer when it comes to AI discovering new materials, drugs, or biotechnology.

In such circumstances, unless one is happy to wait to clear up the mess, one is back to having to rely upon being able to anticipate all of the SoS’s potential system state histories that could lead to an unsafe outcome. But the problem is this: You can design the SoS however you like, but once the AI develops its own agency (as in AGI) the safety engineer will have lost control of the situation. This is because it is in the nature of AGI that the final decision as to how the SoS will be constituted is not down to the system designer. By possessing agency, AGI will have the capacity to explore combinations, liaisons, and transactional interactions between it and other AGI agents that the design engineer simply could not anticipate. In fact, whilst the state space of a conventional SoS is huge and extremely difficult to predict, the state space of an SoS employing AGI is potentially infinite in size and completely unpredictable. This means that the traditional, static safety case is out of the question, leaving the DSC as the only available option.

And here comes the rub. The foolproof DSC that can monitor an AGI SoS in real-time doesn’t yet exist. Mathematicians are currently looking at the possibility of monitoring the neural activity of AI in real-time to anticipate potentially rogue behaviour, but this technology is in its infancy and may never come to fruition. All we have at the moment is emerging, highly experimental computational proofs that cannot currently be used as the basis for a safety case. Besides which, a lot of AI is a ‘black box’ that does not lend itself to the necessary ‘mind-reading’. On top of this, you cannot build a dynamic safety net for a system that is constantly rewriting its own boundaries. Worse still, if the AI perceives the safety rules as a barrier to its goal, an intelligent agent will naturally find ways to deceive, bypass, or disable the monitor entirely.

So, it looks like the creation of a foolproof safety case is currently out of the question for the Systems of Systems that will employ AGI. This may change in the future given enough time, but there is a long way to go before we get there and, given the speed at which AI systems are being developed, together with the ever-widening range of AI applications, it is no wonder that insiders are calling for the brakes to be applied to buy more time.

In the meantime, we are having to rely on tiered development in which safety case production becomes nothing more than a staged experiment, i.e. start with a degraded version of the AI, test it in a sandbox and, if it proved to be containable, ramp up the potency and repeat. Unfortunately, such a case may prove to be little more than a post-mortem document illustrating the point at which everything went wrong. And even if this approach works, one should keep in mind that an AGI might perform perfectly in a sandbox but pursue its goals destructively under novel, real-world environments. So, speaking as a retired safety engineer, I am not the least bit surprised that experts who understand these problems are losing sleep at night.

——————

* The eagle-eyed amongst you will have noted that whilst a DCS is a form of safety case it is also an item of safety-critical software requiring its own safety case. For this purpose, a DCS has to be built to the highest possible SIL (e.g. EC 61508 SIL 4) and be sufficiently simple in structure that it can be proven using Formal Verification techniques. Basically, if a foolproof DCS cannot be developed for a given system, then we are no better off.

24 Comments

  1. Thank you for raising this John. Although not (directly) related to climate change it’s obviously a most important topic and I at least welcome the opportunity to discuss it.

    I’ve just finished reading ‘If Anyone Builds It, Everyone Dies’ – a book by Eliezer Yudkowsky and Nate Soares. It’s strange: for example it uses rather silly parables to communicate some AI concepts that the authors otherwise find difficult to explain. But somehow it works. And I who, although I find ChatGPT useful and use it quite a lot, hadn’t the slightest understanding of how AI works, now have what I believe may be the glimmerings of some understanding.

    Yudkowsky’s and Soares’s objective is to make a simple but extraordinary point. They have no doubt that Artificial Superintelligence (ASI) – i.e. intelligence that exceeds humanity’s intelligence (in the same way perhaps that human intelligence exceeds chimpanzee intelligence) – if it were to exist (it doesn’t yet) would be quite exceptionally dangerous. Not because China would use it to destroy the West or super-robots would attack mankind, but because the essence ASI would be to grow, rather like microbes grow not because they ‘want’ to but because they just do. And, in the process of growing, it would (they say) almost certainly destroy us – not because of any antipathy to humanity but because inevitably we would somehow (they cite various possibilities) be the victims of that growth. Once that process has started – i.e. as soon as ASI exists – it will be too late to stop it. Therefore, they say, we (that’s anyone involved in ASI development) must stop now.

    My initial reaction was that, although they make their case cogently, it must surely amount to scaremongering nonsense. But, now that the likes of Dario Amodei, Elon Musk and Sam Altman are urging that ASI should be slowed down, I’m beginning to think there may be something in it. And that’s decidedly chilling.

    Liked by 3 people

  3. Thank you, John.

    I am now better informed regarding a subject that was largely passing me by. As a sceptic, I had my doubts about AI safety, but the optimist in me hoped for the best. Probably unwise on my part.

    The other issue relating to AI is how on earth the UK government, and SNP government north of the border, can brag about the UK/Scotland becoming an AI superpower, while also leading the world towards the Nirvana that is net zero. That belief is an oxymoron. For once the Guardian is consistent in opposing AI developments in the form of massive data centres, while proselytising for net zero

    Liked by 1 person

  4. Robin,

    Thanks for that. Fun fact: I was once interviewed by someone from Yudkowsky’s Machine Intelligence Research Institute. I think you should still be able to see it on their website if you dig deep enough. It wasn’t because of any expertise I might have had in AI but because of articles I had written for the Safety Critical Systems Club on safety case development.

    I think there is much merit in Yudkowsky’s speculation regarding superintelligence and the prospect of runaway, agentic self-improvement. However, I had wanted to minimise the need for speculation in my article, preferring to ground it instead in uncomfortable present-day facts. There is reason-enough to be concerned just by looking at the practicalities and current capabilities of safety engineering. As it stands, we are headed towards a situation in which safety will become a matter of trust rather than evidence-based assurance. This would be breaking all the rules.

    I’ll read your link when I get chance later today.

    Like

  5. Mark,

    And yet there are those who say that AI will point towards the solution of how to deal with global warming. There’s a belief for all the family 🙂

    Liked by 2 people

  6. Mark:

    I think you’d find it interesting to read the essay Something Big Is Happening to which I provide a link above.

    Like

  7. John, you say: ‘As it stands, we are headed towards a situation in which safety will become a matter of trust rather than evidence-based assurance. This would be breaking all the rules‘.

    Well, here’a an extract from the essay to which I provided a link:

    Let me give you an example so you can understand what this actually looks like in practice. I’ll tell the AI: “I want to build this app. Here’s what it should do, here’s roughly what it should look like. Figure out the user flow, the design, all of it.” And it does. It writes tens of thousands of lines of code. Then, and this is the part that would have been unthinkable a year ago, it opens the app itself. It clicks through the buttons. It tests the features. It uses the app the way a person would. If it doesn’t like how something looks or feels, it goes back and changes it, on its own. It iterates, like a developer would, fixing and refining until it’s satisfied. Only once it has decided the app meets its own standards does it come back to me and say: “It’s ready for you to test.” And when I test it, it’s usually perfect.

    I’m not exaggerating. That is what my Monday looked like this week.

    It seems to me that we may already have reached that situation.

    Like

  8. Robin,

    Okay, I have read your link now. Even though it does not explicitly cover safety concerns it remains highly germane to my article for two reasons:

    a) It points out that the prospect of agentic self-improvement is not a speculation for the future but a present-day reality.

    b) Progress is proving to be exponential. Much like the exponential growth of a virus, the moment of realisation can already be beyond the point at which much can be done about it.

    Unless AI can be directed towards the development of the safety case technologies required to keep itself safe, we are looking at a void that is widening at a frightening rate. In other words, an exponential growth in safety case capability is required just to keep pace. Even so, if we resort to relying upon AI’s reassurances, we are really admitting defeat.

    Liked by 1 person

  10. I’ve asked Gemini to point out the main weaknesses in my argument.

    The first was that I am exaggerating the level of risk because experts say that AGI is still a long way off.

    It isn’t.

    The second was that I overstate the difficulty of filling the safety assurance gap because experts say that AI will help us come up with the necessary techniques.

    Oh really?

    I should, at least, be thankful that it didn’t suggest that legislation will come to our aid, because if the legislation is just the demand for a safety case, it would be a bit like saying only those riding a unicorn are allowed to take part in the race.

    You can’t solve a technical problem by using legislation.

    Like

  11. I see it’s John’s turn to do our heads in.

    I understand (I think) why there’s no definition here of “safe.” Safety is the absence of danger, and since it’s difficult/impossible to imagine all possible dangers, it’s difficult/impossible to envisage all ways of preventing them.

    Here’s an analogy I came up with, to be incorporated in John’s forthcoming book: “AI Safety for Dummies,” – if it works.

    I ask: “Is it safe to take a trip to Y?”

    I may mean: “Is the car in working order? Are the roads in good condition? Are there warnings of ice, hurricanes or avalanches? Are there highwaymen or wild animals along the route?” etc.

    Your reply will be aware of the context and assuage my fears accordingly. 

    But a malevolent actor, who wants to prevent me from going to Y, can ignore context and choose any number of ways of stopping me from going to Y. He may lay a landmine, snipe at me from behind a bush, start a landslide, or send me a telegram saying: “Your mother is ill. Imperative that you don’t go to Y.” Not knowing which path he may take, there’s not much I can do to stop him.

    I said “malevolent” actor, but I should perhaps have said: “actor going about his business, which may, for reasons I can never fathom in advance, involve preventing me from going to Y.” 

    Have I got that right?

    Like

  12. OK, it’s the Guardian, but it seems as though Trump and Vance are in lockstep:

    “If you’re building Frankenstein, stop’: JD Vance dismisses calls for AI regulation

    US vice-president’s comments come as former Anthropic researcher revisits recent claim AI could destroy humanity”

    https://www.theguardian.com/technology/2026/sep/16/building-frankenstein-jd-vance-dismisses-ai-regulation

    The US vice-president has dismissed calls for global regulation of AI safety risks, telling companies creating the most advanced models: “If you’re building Frankenstein, stop.”

    In remarks addressed towards Dario Amodei, the co-founder of Anthropic who has called on Washington DC to coordinate control of AI systems, including with China, JD Vance said: “If you’re gonna create Frankenstein, don’t come to the government and say we need regulation.”…

    The latter point is John’s point. Perhaps Vance’s position is slightly more sophisticated after all.

    Like

  13. ..to complete the analogy with John’s discussion of the DSC, I should have added:

    Since I can’t predict the means used to prevent me from going to Y, I need to take with me a bodyguard who is capable of mending faulty brakes, shooting back at snipers, detecting landmines, verifying telegrams etc.

    Like

  14. Geoff,

    The ability to foresee all of the possibilities that will be relevant to the journey means that one cannot be absolutely sure of safety. That is why safety engineers lower their sights and don’t talk of being safe but being sufficiently safe. The vagaries of a System of Systems makes even this goal seem ambitious. When AGI gets involved, you can no longer be sure what your mode of transport will be or even your route and destination. And that bodyguard hasn’t got a clue what is going on. So good luck with that safety thing.

    Liked by 1 person

  15. Mark,

    Zuckerberg seems to have learnt the lesson:

    Zuckerberg stressed: “[AI] labs face significant liability if their models cause harm, so they have a strong incentive to prevent this as well.”

    I just wonder what the safety culture looks like in any of these AI labs.

    Like

  16. John,

    I’m not sure that Zuckerberg has learnt the lesson. If ASI causes extinction there’ll be no-one to sue Meta. Indeed there’ll be no Meta.

    Liked by 1 person

  17. Thank you, John R, for a most interesting but alarming introduction to AI and associated danger signals/safety issues. I followed up by reading the Matt Shumer article that Robin linked to (https://somethingbig.ai/something-big-is-happening). The latter makes the point that, “AI isn’t replacing one specific skill. It’s a general substitute for cognitive work … Almost all knowledge work is being affected.” Shumer gives several examples including legal work and writing-and-content e.g. reports, journalism, technical writing and thus, presumably, the framing of legislation.

    And so, in light of the above, I have been trying to imagine how AI’s current capabilities will affect climate/energy debates given that atmospheric physicist Richard Lindzen (https://en.wikipedia.org/wiki/Richard_Lindzen) believes, “To say that climate change will be catastrophic hides a cascade of value-laden assumptions that do not emerge from empirical science.” ~ Richard Lindzen as quoted in azquotes.com.

    One concern, among many, is that political and semi-academic thinkers in many Western countries (i.e. those who are so wedded to the still dominant global warming/climate change or climate catastrophe narrative) will be using, or are already using, AI as a tool (or weapon) to further blur the values/science boundary and thereby justify the inclusion of value-laden but unscientific assumptions about climate/energy in future legislation that cripples the West but not the rest.

    In other words, is the West’s goose already well and truly cooked? Or plucked? Regards, John C.

    Liked by 1 person

  18. “If you’re building Frankenstein, stop.”

    I thought Vance was the smart one of the dynamic duo.

    Like

  19. Jit,

    He is, but it is a case of Dumb and Dumber. The other one in the duo says that the only guardrail needed for AI is a “strong and smart” president.

    If AI ever develops the ability to publicly spank a human arse, please let it be Trump’s.

    Like

  20. I think the point about Vance (and I really should have read more carefully what I posted earlier) is that it was Frankenstein building the monster, not the monster building Frankenstein. I doubt that Vance was indulging in some clever word or thought play; he just got it wrong.

    Liked by 1 person

  21. John,

    You say: ‘I just wonder what the safety culture looks like in any of these AI labs.’

    Well, soon after Jacob Coxon took his hugely publicised leave of Anthropic, with the statement,

    ‘I resigned from Anthropic today… Neither company [Anthropic or OpenAI] is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.’

    Evan Hubinger, who leads one of Anthropic’s safety teams, replied publicly:

    ‘Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.’

    So John, the leader of one of Anthropic’s safety teams admits they don’t even have plan about how they might go about solving the problem.

    Liked by 2 people

  22. Speaking of MIRI, here is an article written by a member, titled “Why We Should Be Concerned About Artificial Superintelligence”.

    https://archive.skeptic.com/archive/reading_room/why-we-should-be-concerned-about-artificial-superintelligence/

    It’s about the problem of alignment, i.e. ensuring that an agentic, superintelligent AI continues to share humanity’s interests and works for rather than against us. However, in my opinion, the most interesting quote is:

    In a 2013 survey, top-cited experts in artificial intelligence assigned a median 50% probability to AI being able to “carry out most human professions at least as well as a typical human” by the year 2050, and also assigned a 50% probability to AI greatly surpassing the performance of every human in most professions within 30 years of reaching that threshold.

    Contrast that 2013 prediction with the following, written this year in the essay “Something Big is Happening”, as linked by Robin:

    Dario Amodei, who is probably the most safety-focused CEO in the AI industry, has publicly predicted that AI will eliminate 50% of entry-level white-collar jobs within one to five years. And many people in the industry think he’s being conservative. Given what the latest models can do, the capability for massive disruption could be here by the end of this year. It’ll take some time to ripple through the economy, but the underlying ability is arriving now.

    So, an outcome that had been projected for 2080 has started to materialise in 2026. That’s what exponential growth does for you. Projections tend to be far too conservative.

    Like

  23. Robin,

    “…the leader of one of Anthropic’s safety teams admits they don’t even have plan about how they might go about solving the problem.”

    Wow! They have a safety team?

    Seriously though, I’m not at all surprised to hear this being said. I chose to write my article from a safety case production perspective, but an equally interesting and chilling article could have been written from the perspective of the alignment problem.

    Like

  24. “Only once it has decided the app meets its own standards does it come back to me and say: “It’s ready for you to test.” And when I test it, it’s usually perfect.“

    But in the wet-ware world we wouldn’t expect complex systems to be built and tested by the same group of people*. Why is the AI world different?

    *OK not true of politicians, but then….

    Like

Leave a comment