Climate AI models: nine researchers call for a physical assessment, not just forecast scores

Weather and climate models are being developed faster than our capacity to evaluate them. The authors propose five priorities and three areas of focus for the coming decade, including a programme dedicated to their reliability.

Pre-trained on weather and climate data, foundation models are increasingly adapted to uses in Earth sciences well beyond weather forecasting. In an article published on arXiv, Makoto Kelp and eight co-authors warn that their adoption is faster than the scientific community's ability to evaluate them, and they are judged almost solely on performance scores.

Good forecasting does not mean good physics

These scores measure the difference between a prediction and a reference product, but not how the model represents physical processes. This difference becomes critical in a changing climate, i.e. in conditions that the models have never encountered during their training.

The authors identify five priorities for physical evaluation (training data, fine-tuning, behavioral testing, mechanistic interpretability, validation of outputs) and recommend open evaluation datasets, a common reporting standard and a research program dedicated to the safety of these models for the coming decade.