AstroBridge teaches an LLM to read directly spectra, images and light curves, and distinguishes quasars and galaxies at 91.5%

To link astronomical observations to language, AstroBridge automatically matches data and passages from scientific articles. The model obtained increased the distinction between quasars and galaxies from 69% to 91.5%.

Astronomical observations take the form of images, spectra or curves of light, while the scientific literature describes these same objects in natural language. An international team, including Ibrahim Habib, Joshua Speagle and Kartheik Iyer, proposes AstroBridge, a framework that bridges the two worlds by directly connecting this data to a large language model.

The trick is to do without manual annotation. The metadata of the catalogues makes it possible to associate each observation with scientific texts that talk about the same object; a vision-language model then writes a legend from the observation, the passages found and the available information. AstroBridge then learns to produce these legends from the raw data alone: fixed encoders, specific to each type of observation, extract representations that a trainable fusion network aligns with the LLM space.

On the spectra, the model distinguishes quasars and galaxies with 91.5% accuracy, compared to 69% for the basic model before adjustment. The authors acknowledge weaknesses in the morphology of galaxies, in the rare classes of supernovae and in the factual reliability of the legends generated, but see this as a first step towards a pooling of all astronomical data through text.