Can foundational models in astronomy predict when they are going to make a mistake?

A study uses predictive validation to test the reliability of representations of celestial objects when switching between different telescopes: the Legacy Survey, HSC, SDSS and JWST.

Foundation models on images from the sky increasingly serve as a common basis for analyzing data from different surveys. But is the same object represented consistently from one instrument to another, and can the model indicate in advance that the transfer will fail? A team of five researchers proposes a framework based on consistent prediction to audit it, based on objects observed by both the Legacy Survey and HSC, by HSC and the James Webb Space Telescope, or by SDSS and HSC.

Sometimes silent failures

The representations of the specialized model AstroPT translate better from one survey to another than those of generalist models of vision, and each family of models fails on different objects. An alert signal from the source representation works well for the Legacy to HSC transition, depends on the model for HSC to JWST and is almost missing for SDSS to HSC. The authors also note errors that are not explained by the brightness or the apparent size of the objects, and conclude that the consistent uncertainty allows us to identify when a representation is reliable and when it fails without warning.