As a former software engineer, turned quality assurance practitioner, turned functional safety analyst, and now basking in my retirement, I find myself with both time and reason to reflect. And when those reflections overlap with concerns for climate change, there is one inevitable question that keeps cropping up: Do climate modellers apply anywhere near decent standards in their software development? There is both a short and a long answer to this question. The short answer is ‘no’. Here is the long answer.
Some context
Before I get into detail, I should point out that the problem of inadequate standards is by no means restricted to climate science. It is, in fact, applicable to most scientific research software, and it stems from the attitudes and priorities that exist within the scientific community as a whole. The problem is so well-known that its impact has been allocated its own jargon; it is known as technical debt.
In due course I will fully explain what is meant by technical debt, but first I would like to illustrate by offering a notorious example. And in order to emphasise that technical debt is not a problem exclusive to climate scientists, the example I offer comes from the world of epidemiology.
During the Covid pandemic, a great deal of government policy relied upon predictions made by epidemiological models. Specifically, the UK placed great store in a model known as CovidSim, developed by Neil Ferguson of the Imperial College London. However, when the source code for this model was finally made public on GitHub in mid-2020, the wider software engineering community found it to be wanting.
Actually, the technical term used by many of that community was ‘horrendous bag of crap’. Firstly, it was a monolithic file of over 15,000 lines of code, enjoying the structure of a plate of spaghetti bolognese. Upon detailed examination, a total of over 1,200 ‘issues’ were documented, not the least of which was the lack of reproducibility. Basically, different outputs were produced depending upon the platform upon which it was run. The indeterminism isn’t a bug, it’s a feature, opined Ferguson. We don’t care, replied the software engineers; a lack of determinism is the last feature you would want in software that would be used to draft sweeping national laws destined to cause grievous damage to the economy.
In light of these problems, a rescue mission was launched in April and May of 2020. Software engineers from Microsoft and GitHub, and the odd video game programming legend, volunteered to restructure and refactor the code. An independent peer-review group called Codecheck later verified that the heavily cleaned-up, open-source version was now stable and deterministic. Problem solved, but at a not inconsiderable cost when measured in terms of lost reputation and remediation effort. It was a classic example of technical debt.
The point is this: Ferguson could have avoided all of this had he placed greater importance upon developing his code to widely-recognised, professional standards. He could have made it modular, well-documented and thoroughly tested to eradicate bugs. But he didn’t. And the reason was because he was initially developing a tool for his own scientific research. It wasn’t necessary for the software to meet the standards expected of commercial software — only that it should provide him with the wherewithal to pursue his scientific objectives. Such corner-cutting seems perfectly reasonable, until the goalposts shift and public scrutiny, wider applicability, maintainability and high levels of required integrity become important issues. It is at that point that the corner-cutting has to be paid for. That is what is known as technical debt.
As I have said, this potential problem exists throughout the world of scientific software. Scientists are always focussed upon what is required to fulfil the goals of scientific research; all other goals will be an afterthought at best. One can express this in terms of the quality assurance concepts of validation and verification. Scientists are very concerned that their software models should be valid, i.e. serve the required purpose. This can be determined simply by scientifically evaluating the outputs. What they are not so hot on is producing software in a manner that facilitates the verification of its correct development. This is a problem because validation without verification is a recipe for disaster. Without sound verification, the validity can be an illusion.
So, what about climate models?
Climate modelling is not miraculously free from technical debt. On the contrary, the scale of the problem is such that no one involved could dream of ignoring it. This is not the stuff of conspiracy theory; climate scientists and computational researchers have produced numerous peer-reviewed papers, hosted endless workshops, and participated in formal studies that highlight the extent of their technical debt. The problems are not difficult to diagnose.
Firstly, there is the problem of the ‘Fortran legacy’. Climate models were first developed back in the time when Fortran was all the rage (a language invented probably before you were born). To be fair, it was ideally suited to the problem. Consequently, even state-of-the-art global climate models still consist of millions of lines of far from state-of-the-art programming that has been passed down and modified across decades – often involving shortcut hacks. Evidence for this can be found in the so-called Self-Admitted Technical Debt (SATD) that takes the form of thousands of IOU-style code comments along the lines of ‘Sorry, I have put this ugly patch in here because I didn’t have the time or confidence to properly refactor this horrible legacy code’. I sympathise with this because I have done it myself in my own software engineering career. It’s expedient, but it just adds to the debt mountain.
Meanwhile, for the new generation of climate scientists, a program written in Fortran might as well be written in Latin. If they have any software training at all, it would not be on how to produce and maintain millions of lines of opaquely complex and entangled code written in an ancient tongue. They might know better, but they are stuck with what they have got and the technical debt involved in sorting this mess out is not to be underestimated. The community openly discusses the immense difficulty of remediation; as one observer has put it:
Re-writing a codebase with over 9000 commits of Fortran is like tearing down and rebuilding a house. Re-writing a climate model is like rebuilding the entire neighbourhood.
However, let’s face it, millennial climate scientists will not have been plucked from the ranks of fully-trained software engineers, let alone the embattled community of software quality assurance professionals. They will likely be accidental programmers that know just enough to get by whilst they pursue their first love – the science. What use have they for ISO9001 with TickIT when they have a model that is churning out the sort of predictions that their scientific livelihood thrives upon? But as I said earlier, validation without sound verification can trip you up. Many climate models are opaque ‘black boxes’ where it must be hard to isolate whether an unexpected output is a breakthrough in physical theory or a silent floating-point rounding error.
However, before you get too excited, I should point out that none of this is reason enough to dismiss climate models as bug-ridden crocks to be avoided at all costs. In fact, there are good reasons to expect that climate models may have a lower defect density than most other programs. In effect, they are open-source software in which code has been passed about so often, and scrutinised by so many peers, that it becomes difficult for bugs to survive. And you can say what you want about climate modellers, they are rather clever people who are highly motivated to construct their tools properly. Furthermore, it should be acknowledged that there is a lot going on to fix the debt.
Where is the community going from here?
Much as the climate modelling community might wish to continue living off its technical debt, it can’t do so forever. The legacy code was written to execute on your average CPU, but that is not how things have remained. Modern-day climate science modelling demands a lot of compute utilising the parallel processing power of the GPU. The time to call in the debt has arrived as code is re-written to execute on fundamentally different architectures. Basically, climate science is moving house, so it is now or never for getting its house in order. Consequently, a lot of effort is now being put into re-writing software just to deal with the hardware advances, without there being any actual advances in the science. I’m afraid that is what technical debt payback looks like.
A good example of cutting-edge modelling, designed to solve the architectural limits of older models, is the Icosahedral Nonhydrostatic Weather and Climate Model (ICON), developed jointly by the Max Planck Institute for Meteorology (MPI-M) and the German Weather Service (DWD). The model was originally written in Fortran but, instead of just continuing to patch old code, the ICON open-source community joined with Swiss supercomputing centre CSCS to rewrite and port the model’s dynamical core to GPUs using OpenACC directives and Domain-Specific Languages (DSLs). This vastly improved maintainability because the same code can now run on either CPUs or GPUs without requiring scientists to write two entirely separate versions of their physics equations.
Another initiative well worth mentioning is the US National Center for Atmospheric Research (NCAR) Common Infrastructure for Modeling the Earth (CIME) framework. NCAR had a substantial technical debt in the form of the Community Earth System Model (CESM) that glues together massive, independent sub-models (atmosphere, ocean, land, sea ice) written by different scientific teams over 30 years. It consists of millions of lines of mostly Fortran code produced in a variety of styles in accordance with various legacy standards. CIME helps to improve maintainability of this code by separating the ‘scientific code’ from the automation/build code. CIME handles the compilation, testing, and system-level setups, thereby removing the burden of software engineering from the climate scientists and leaving them to deal only with the bits that tickle their scientific fancies.
These and other developments are no doubt helping to pay back the debt, but the scale of the problem is daunting. The sad fact is that, in keeping with most other scientific research programming, climate modelling is where good software development practice went to die. In a scientist’s hands, software is just a tool, not a product to be built to a standard. It is often highly experimental; potentially disposable and hence unworthy of future-proofing. It is said that the proof of the pudding is often in the eating; if it is good enough to generate scientifically plausible results, then it is good enough. Little more is asked of it. Climate model programming followed that groove, but it really shouldn’t have. Climate models are as much a product as they are a tool, and they sit at the centre of political strategies that will cost trillions of dollars, should they prove to be flawed. By running up a technical debt, the community developing these models created a ticking time bomb, destined to go off in their communal face, much like CovidSim did for Professor Ferguson.
When the CovidSim debacle hit the streets, the British Computer Society (BCS) was amongst those that called for any scientific research code that had potential safety and national security ramifications to be written to recognised software development standards. I took the BCS’s concerns with me to an online forum comprised mainly of practicing scientists, congregating at the blog And Then There’s Physics (ATTP). The response was hostile and dismissive: The BCS clearly didn’t know what it was talking about. It was a joke and had no idea what scientific software development entailed. Its call for costly standards was entirely unnecessary and could only do harm. Scientists know what they are doing and software engineers and quality assurance time-wasters should keep their advice to themselves. It’s all there for you to read.
All I can say is that I am pleased to see that the climate scientists themselves seem a lot more cognisant of the technical debt, and far more receptive to the idea of repaying it than are those fellow scientists who would rush to their defence.
Further Reading
The Nature of Technical Debt in Research Software
Multi-Artifact Analysis of Self-Admitted Technical Debt in Scientific Software
Tackling Tech Debt in Scientific Research
Climate Models: Challenges for Fortran Development Tools
Why are Climate models written in programming languages from 1950?
How Climate Model Developers Deal With Bugs
Climate Models are Good Quality Software (?!)
Cracking the code: Linking good modeling and coding practices for new ecological modelers
What are the biggest challenges and innovations for new climate models?
CESM Tutorial – Introduction to CESM2
Operational numerical weather prediction with ICON on GPUs (version 2024.10)
Coding that led to lockdown was ‘totally unreliable’ and a ‘buggy mess’, say experts
The Software that Led to the Lockdown
Codecheck confirms reproducibility of COVID-19 model results
BCS calls for computer coding in scientific research to be more professional
By the time CovidSim was released on GitHub it was still plagued with lack of structure, global variables, no unit test framework and two newly implemented regression tests. The derogatory phrase “Fortran written in C/C++” was appropriate. (Note that modern Fortran allows proper structure but does maintain backward compatibility.) Whether the code actually implemented the model from the published papers or a bug ridden approximation is up to speculation. Given that it’s modelling an epidemic there are concerns about whether the model is quantitative or qualitative (aka a toy model).
A couple of years ago I downloaded the Fortran source code of NASA GISS ModelE. It included labelled DO loops that were superseded by the 1978 DoD extensions and can be easily replaced by about 20 lines of script code if the maintainers weren’t so feckless. Loads of GOTO statements in defiance of the 1968 letter of Edsger Dijkstra. However my main interest was the stochastic modelling and there I found arbitrarily scaled uniform noise being added into the temperature field. As far as I can tell that is completely unphysical.
LikeLiked by 2 people
I don’t think it’s worth trying to simulate the entire Earth in order to estimate feedbacks. Last time I checked (the previous generation) the range of possibilities modelled ranged from “no danger” to “apocalypse now.” I would actually have more faith in a one-line equation than in something with a million lines of code modelling an atmosphere based on 100 km diameter slabs 1 km high with a time step of 30 minutes. [I appreciate this says nothing about how the model is constructed code-wise.]
Meanwhile, I hear OpenAI has “solved” Navier-Stokes, so perhaps we should leave future climate states to AI to model.
LikeLiked by 1 person
cheerfullynoisy24384a8eb5,
“Fortran written in C/C++”
Yes, I’m afraid that is a thing 🙂
LikeLike
Jit,
“I would actually have more faith in a one-line equation than in something with a million lines of code modelling an atmosphere based on 100 km diameter slabs 1 km high with a time step of 30 minutes.”
It’s funny that you should say that. This is what Nate Silver has to say in his book The Signal and the Noise regarding climate model complexity:
LikeLiked by 2 people
Since you mention aTTP, coincidentally I wandered over there the other day to see what’s going on. Not a lot, is the answer. Only two articles this year, and only one of those was vaguely related to climate, an attempt to downplay the discrediting of RCP8.5. I don’t think it will age well.
LikeLike
One factor in the problem is that the profs who want the software developed will generally hand it to the most clueless of their post-docs and tell them to get on with it. They have no clue about structured programming and neither does the author of the code, with the results that we (all too often) see.
The really stupid thing is that their uni will almost certainly have a CompSci department who ought to have plenty of people who could do a proper job, given a proper spec, and might welcome a ‘real world’ project to work on.
LikeLiked by 1 person
Made me look – funny comments from 2020 –
“Everett F Sargent says: June 4, 2020 at 6:01 pm The irony is just killing me …
Computer C-O-D-I-N-G in scientific research must be professionalised to restore trust – BCS
https://www.bcs.org/more/about-us/press-office/press-releases/computer-coding-in-scientific-research-must-be-professionalised-to-restore-trust-bcs/
“The computer C-O-D-E behind the scientific modelling of epidemics like COVID-19 should meet independent professional standards to ensure public trust, according to BCS, The Chartered Institute for IT.”
AFAIK C-O-D-I-N-G equals software development.
The C-O-D-I-N-G police bureaucracy would stop all software development in its tracks, so I would suggest them going after Microsoft first. Then Facebook. Then Google. Then Twitter. Then … ad infinitum ad nauseam.”
Plus Willard is not fooled – “He’s supposed to be big in risk management, but don’t sweat it”
And as usual not sure what Mosher means/is trying to say!!!
LikeLike
Last quote from me from Mosher which rang some bells –
“standards will evolve and be adopted, standards that arise organically from the practice tend to A) be better, B) be more readily accepted. But you need incentives.
Bottom line, if folks want better code (systems) from academics then you better be willing to pay
and make investments and reward people accordingly. It isn’t going to magically happen
by chanting “open code” and “ISO” ( I know I tried)”.
Yeah you did Mosh, I remember it well from your Climate Audit days. Took a leak to get some emails, remember the Harry IT guy saying the model code was crap.
LikeLike
It seems to me that objections to climate modelling are higher level than this. I come from a background of creating and running models of reservoirs in the oil and gas industry. Ironically, most of the modelling techniques and statistics used in what is called climate science are taken from my industry.
To model any physical phenomenon, three things must be true. You must be able to identify the things that are causing the thing you’re interested in to vary. For each of those things, you must have a sufficiently accurate physical description. And that whole system has to be computationally representable.
None of those conditions are met in climate modelling. They can see that the climate is varying in response to phenomena that they can’t label. Many of the phenomena they can label, they can’t describe physically – for example, the microphysics of cloud formation. And coupled nonlinear systems like the climate are computationally non-representable.
So since they can’t model the system, what they produce instead are toy models of it based on overfitted statistical correlations. These have the property of being arbitrarily accurate for the historical period over which they’re calibrated. You can construct a statistical model that accurately predicts the length of women’s skirts from the price of cheese over a historical period. But, being unrelated to observable reality, they yield predictions with no physical meaning – neither “right” nor “wrong”, but “not even wrong”, because they fail to meet the minimum criteria by which rightness and wrongness can be judged.
LikeLiked by 1 person
It all comes down to translation. Pure mathematics is the language we human beings developed in order to translate the wonderful, mysterious inner workings of Mother Nature. Her deeds became handed down to us in bite-sized mathematical formulae. Which worked great, for a while, during the 18th-20th century heyday of modern physics and chemistry. But then Babbage went and invented the computer and we needed to translate the translation so computers (which worked in numerical bits – ‘0’ and ‘1’) could understand mathematical formulae. Hence FORTRAN – literally ‘Formula Translation’. Numerical weather models do a reasonable job of translating Nature’s inner workings into useful forecasts a day or two in advance. Alas, Nature has been largely lost in translation with the climate models, and it definitely got lost in Pantsdown Ferguson’s garbled CovidSim translation of the future dastardly deeds of an alleged zoonotic cold virus, if indeed faithfully translating Nature was the principal objective of the attempt to model the ‘Covid Pandemic’, which I doubt.
LikeLike
Dfhunter,
I think there were two basic problems with the ATTP discussion.
First, they were a group of academics and scientists that assumed they were dealing with a knuckle-dragging climate denier. Hence the personal jibes regarding my lack of expert authority.
Secondly, there was a lot of wasted words that stemmed from a failure to understand that calling for better coding standards was actually a call for better standards of coding. Coding standards would certainly help in this respect, but the greater issue was the need to develop software in accordance with a recognised software development life-cycle. Scientific programmers tend to be what software engineers refer to as ‘hackers’, i.e. they can’t stop themselves from going straight to hacking code without any of the preliminaries that would ensure that they do a good job of it. Lots of reasonable excuses were made for this, based upon lack of time and resources, but one suspects that the real problem is a failure to know any better and a personal desire to quickly get down with playing with the computer. Added to this is the fact that, as soon as another party is employed to do the writing for them, they have to effectively explain to someone else what is required – and they can’t be arsed.
Whatever the case, the basic take-away was how resistant a scientific forum had been to advice that is readily accepted outside the halls of academia. I was prepared for the attacks on my own credibility, because that is what one always got when visiting ATTP, but I was not prepared for the asinine attacks on the BCS. I think that this shows that paying back the technical debt may require a huge change of mindset within the scientific community.
LikeLiked by 1 person
John R; thanks for posting this. You do a good job of making comprehensible something which is horrendously complicated (to this computerphobe). A side benefit was to read about the shenanigans around Prof Pantsdown. At the time I found it utterly unbelievable that his “predictions” were listened to, given his abysmal track record and initial refusal (aiui) to share his code and data. If only they had consulted the professionals!
On simplified models, Willis Eschenbach has posted some articles on WUWT, such as:
https://wattsupwiththat.com/2025/12/18/a-spherical-cow-climate-model-that-actually-works/
That may be simplified to him but it went completely over my head! So, if it is not relevant to this discussion, apologies!
LikeLiked by 1 person
Dfhunter,
Also, contrary to what the good folk of ATTP might think, this is actually one of those subjects that I feel eminently qualified to talk about. This is not because I have experience of writing research software (let alone any climate modelling), but because I have first-hand experience of trying to deal with the technical debt caused by a research scientist. The individual concerned was a mathematician employed by the Royal Aircraft Establishment – a person who would generally be referred to as a ‘boffin’. He had written a program that was used to control a 160-ton antenna that was tracking satellites as they crossed the sky. During the traverse, there would two-way telemetry between the ground and the satellite. I can’t say too much, but these were US military satellites used for global surveillance. The problem was that the computer upon which the software ran was obsolete. My employer won the contract to upgrade the system, entailing, amongst other things, the transcription of the software from legacy Fortran to Pascal.
The problem was that the Fortran was written to a typical research scientist’s standard, i.e. it was appalling: Absolutely no accompanying documentation, totally unstructured, riddled with GOTOs, not a single in-line comment, variable names that were all simply single letters of the alphabet, and every other bad practice you could care to mention. He would even use the same variable name for totally different purposes depending upon where it featured in the program — and you had to work out that this was happening for yourself! In a word: it was impenetrable and hopelessly unmaintainable dross. One by one, my fellow software engineers resigned, since they knew that someone would ultimately have the thankless task of making the upgraded system work on site and they didn’t want to be that guy. That person ended up being me. I’ll spare you the details, but a system commissioning that was scheduled for two weeks ended up taking 8 months – that’s me, newly-wed and 8 months away from home working on my own.
One of the very many low points of this episode in my life occurred when I went to see the original author in his Farnborough office, to ask him to talk me through the part of the code that handled the PID control algorithm. He looked at me askance, and said ‘But it is just a standard control algorithm that you can find explained in any textbook’. I carefully explained that I wasn’t looking for a lecture on how such an algorithm worked, but where it was encoded in his program. Unmoved, and obviously unaware of the technical debt he had racked up, he simply repeated his opening statement — and that was that. I was very young and inexperienced at the time, so I just thanked him and retired to cry in my beer. Today, I would have responded by pointing out that if he hadn’t been such an incompetent, lazy, amateurish and unredeemably thoughtless hacker who was too arrogant to countenance the idea that his work was just a crock, then I wouldn’t have the hapless job of trying to turn his shameful programming efforts into something presentable.
In the light of this experience, I have no patience for any special pleading for poor research scientists who fail to see the need to follow good programming standards.
LikeLiked by 1 person
Hi John – a quick link that may be relevant – Don’t “Learn To Code” – Do an A.I. Clone… | Musings from the Chiefio
LikeLike