Chest X-rays: raw accuracy overestimates vision-language models that, out of an abundance of caution, respond with ‘normal’

A test bench evaluates CheXagent and MedGemma across 36 configurations and demonstrates that the choice of prompt and workflow alters the diagnosis, to the extent that a case-by-case referral approach is recommended.

Evaluating a medical vision-language model often comes down to measuring a rate of correct responses on a dataset. A study published on arXiv on October 1, 2026 shows that this figure can be misleading for the interpretation of chest X-rays: a model that conservatively classifies most images as "normal" scores flatteringly without being reliable.

36 configurations screened

The researchers built a "stress test" on two balanced sets, one private and accompanied by reports, the other taken from the public MIMIC database. Three models (CheXagent, MedGemma-4B and MedGemma-27B) were subjected to it with three styles of instructions and two work organizations: a single model, or several agents that reason together. The result: reliability depends strongly on the family and size of the model, and multi-agent reasoning improves some combinations while degrading others.

The team therefore proposes a routing mechanism that decides, at the time of diagnosis, whether it is worthwhile to switch to the more expensive multi-agent mode. It advocates for evaluation protocols that take into account the sensitivity to instructions, the diversity of errors and the workflow chosen before any clinical commissioning.