127,833 requests from healthcare professionals analysed: clinical AI benchmarks do not reflect real-world use

Healthcare professionals are primarily required to carry out administrative paperwork and search for information, with diagnosis accounting for just 3.7 per cent. The median benchmark does not include any documentation tasks.

LLM-based assistants spread in hospitals, and their maturity is most often judged by benchmarks built from carefully selected examination questions or cases. A team wanted to check whether these tests reflected the actual work of these tools, by analyzing 127,833 requests from 6,342 doctors, nurses and practitioners from 35 specialties to the institution assistant for eight months of deployment.

Far from diagnosis

Each request was characterized with RCQ-Map, a grid validated by clinicians that informs the task, intent, ability to respond, missing information and potential risk. Documentation and administrative tasks account for 36.2% of uses, knowledge research 28.9%, diagnosis only 3.7%. More than a third of the questions could not be answered correctly as they were formulated.

Applyed to 58 public benchmarks combined in a Clinical AI Benchmark Atlas, the same grid shows that the median benchmark does not contain any request for documentation and only shares 31% of the division of tasks observed in practice. The suites that are supposed to imitate clinical practice do no better than those of the major lab reports. The authors argue for evaluations based on actual clinical use.