According to a study conducted under a common protocol, AI models miss more extremes when the physical model multiplies false alerts. Each metric penalizes one or the other error, hence conflicting conclusions.
AI-based weather models now compete with physical models for a fraction of the computational cost and are already being used in operations. But studies differ on their ability to anticipate extreme events, and each uses its own evaluation method. Tobias Biegert, Sam Allen, Zhongwei Zhang and Sebastian Lerch compared AI and physical models in a unique experimental setting to understand where these disagreements come from.
Two ways to make a mistake
Their conclusion: the winner changes with the metric. If we only look at the extremes actually observed, the physical model wins; with weighted score functions, it is rather the AI models. Contingency table analysis sheds light on this paradox: AI models too rarely predict extremes, so their errors are mostly missed events, while the physical model announces them more often and accumulates false alarms. Each indicator sanctions these two types of errors differently.
The decomposition of the scores shows that the gap is primarily due to a lack of calibration, and not to a lesser ability to distinguish risk situations. For the authors, this explains the contradictory results of the literature and requires careful selection and explanation of metrics before judging AI models on extremes.
Read the full article on the original source
Go to the source →