Tested on 86 clinical tasks in 20 languages, models that directly attribute a probability to each answer are fast and well calibrated on the text, but pick up on the images and face the triage nurses.
Many clinical decisions boil down to choosing between a few options, quickly and in large numbers, knowing when to hand over to a caregiver. Generative models do not lend themselves to this: they respond slowly, in free text, and misjudge their own trust. A research team therefore evaluated so-called "System 1" models, which give a probability for each possible response without prior examples. Four of them, two proprietary and two open, were confronted with 86 public tasks covering 14 clinical areas and 20 languages, i.e. 179,111 cases in total.
Good in knowledge, poor in visual judgment
Measured by AUROC, the performance of the best model reaches 0.94 in medical knowledge and 0.93 in text comprehension, but falls to 0.84 on 2D image anomalies and 0.69 on volumetric examinations; imaging is on average 0.09 points lower than text. The best model equals or exceeds the best published score on 13 of the 26 textual tasks, compared to 8 out of 35 in imaging. Triage nurses remain clearly ahead (85% correct answers compared to 56%), and accuracy drops to 39 points when the same question switches from English to Yoruba.
On the other hand, the gain in efficiency is spectacular: a decision takes between 0.34 and 0.44 seconds and consumes 24 to 81 joules on an open model run locally, while generative models of the same family are 11 to 76 times slower for only 3 to 5 points of more balanced precision. The authors see it as a tool adapted to the answers already present in a text or in medical knowledge, not to those that must be judged or read on an image.
Read the full article on the original source
Go to the source →