Out of three cases of Korean clinical examination and 7,480 double-blind evaluations, three out of four models remain slightly below the human expert; Llama 3.1 70B picks up on all criteria.
The Objective Structured Clinical Examination (OSCE) assesses the communication skills of future doctors: faced with a "standardised patient", often an actor, the student must conduct the interview and reason clinically. Virtual patients animated by large language models appear to be a practical alternative, but their assessments have so far been based on a single model, small panels or an LLM judge. A Korean team wanted to identify precisely the weaknesses of each model.
A double-blind protocol
Four models (GPT-4o, Claude 3.5 Sonnet, Llama-3.1 70B and the Korean HyperCLOVA X) were compared to a reference produced by a human expert, on three clinical cases: hematuria, fever during pregnancy and chronic cough. Seventeen students at the end of the course noted each line, without knowing who had produced it, on three criteria at the sentence level and six at the consultation level, i.e. 7,480 data.
Small differences, except for one model
At the consultation level, three out of four models are between 0.08 and 0.47 points (on a Likert scale) below the human reference for comprehensibility, coherence and absence of hallucinations. Some differences are not significant: absence of hallucinations for Claude 3.5 Sonnet and GPT-4o, consistency for GPT-4o, commitment for HyperCLOVA X. Llama-3.1 70B, on the other hand, remains significantly below on all six criteria. The authors also present a web platform for training at the OSCE that collects, with consent, dialogues between students and virtual patients.
Read the full article on the original source
Go to the source →