Diagnosis: models make the right call based on insufficient evidence, and the order of symptoms alters their verdict

MedEVM applies nine models to 1,050 cases where evidence trickles in. A safeguard that verifies the evidence before drawing conclusions improves accuracy by between 12 and 51 points.

A good diagnosis obtained by chance remains a clinical risk, but the simple measure of the correct response rate rewards it. Researchers name this gap "Evidence-Value Misalignment" and build MedEVM, an environment of 1,050 cases covering 24 pathological systems, where observations are delivered turn after turn: the model must choose between waiting for other elements or deciding.

Four crossovers on nine models

Models do not judge whether the clues are sufficient, including the most powerful, and the reasoning mode exacerbates the problem. They sometimes delay concluding when they are already leaning for the right answer. Presenting the same elements in another order modifies the diagnosis, and a misleading index added after the blow continues to mislead the model even when the available information was already sufficient.

The team proposes EVD-Harness, which separates the generation of assumptions from the decision to conclude, with an audit of the evidence between the two. In five models, accuracy increases from 12.0 to 51.1 points.