ClimateBench v2: a common test bed for evaluating physical and AI models of the climate in 2050

The protocol treats physical, hybrid and purely data-driven models equally, testing them against ten years of observations from after 2015 and against past climates.

A team led by Duncan Watson-Parris publishes ClimateBench v2, a standardised protocol to assess the ability of climate models to project regional changes in temperature and precipitation by mid-century. The ambition is to compare physics-based models, AI emulators and hybrid approaches on an equal footing.

Three levels of tests

The first level verifies basic physical credibility: energy conservation, coupled variability, response to forcings. The second level compares the models with observations after 2015 (surface temperature, precipitation, radiative fluxes, sea ice, major modes of variability) using a probabilistic score, which requires the submission of several ensemble members. The third tests the generalization outside the distribution, on past climates (last interglacial, last glacial maximum, middle Holocene) and on experiments where an AI model must predict the future of a model of the Earth system based on its history alone.

All observations after 2015 are for testing only: the decade since the end of the historical CMIP6 simulations constitutes, according to the authors, an unprecedented sample of real climate change for the current generation of models. The evaluation code, reference sets and training data are published in open access.