AI Consilium: using LLMs to generate a second multidisciplinary medical opinion, and assessing whether it holds water

A test set comprising 300 resolved clinical cases evaluates simulated ‘consultation meetings’ between models. Structured reports outperform unstructured ones in 7 out of 8 pipelines, with GPT-5.6-Sol coming out on top.

A team led notably by Ivan Oseledets and Dmitry Dylov proposes AI Consilium, a test bench that evaluates the ability of large language models to produce a second medical opinion as would a multidisciplinary meeting: each model plays several roles as specialists, then a final synthesis is written. The objective is to ensure that these opinions are factual, clinically consistent, exploitable and safe.

300 cases, 3,040 reports

The data set brings together 300 cases on the established diagnosis, drawn from NEJM-MedQA: 110 are used to calibrate notation from physician evaluations, 190 are reserved for testing. A total of 3,040 final reports from eight single model pipelines were evaluated on differential diagnosis, management and clinical reasoning.

Main result: imposing a structured format on synthesis improves quality in 7 out of 8 pipelines, especially for management recommendations (2,740 versus 2,436 in open-ended writing). GPT-5.6-Sol gets the best scores and achieves consensus in a minimum of rounds of discussion. The work was tabled on arXiv on October 15 and has not yet been peer reviewed.