Spanish boarding school competition (MIR): Open models would get a specialty, but not always for the right reasons

Of 1,552 questions from eight IRM sessions, three open models of 8 to 27 billion parameters would rank among the first 600 to 3,000 candidates. Their clinical reasoning often fades.

A team from the University of Deusto submitted four open language models, ranging from 8 to 27 billion parameters, to the MIR examination which opens up access to the medical boarding school in Spain. The researchers noted 1,552 questions from eight editions using the official scale. The goal was not only to measure a score, but to answer questions from a health care system trying to run a model on its own servers.

A good trompe-l'oeil ranking

By answering all the questions, the three largest models would place themselves roughly between the 600^e and 3 000^e place, with 41 to 43 accessible specialties. Abstaining does not bring them anything: even their least assured answers are fair more than once in four, the threshold of profitability of the scale. Medical fine tuning reduces their excess of confidence, but at the cost of a decrease in accuracy and without better distinguishing the right answers from the wrong ones.

Above all, 13 to 38% of the correct answers exclude a distractor for a clinically wrong reason, without the level of confidence indicating it. And removing or inverting the images provided with the questions does not change the accuracy: the models obviously do not use them.