Nasopharynx cancer: brilliant diagnostic LLM stumbles on patient follow-up, shows a test bench on 405 files

NPC-RubricBench evaluates 30 models at each stage of support. Medical models forget decisions more often than generalist models.

Most medical AI evaluations focus on an isolated consultation. A Chinese team proposes with NPC-RubricBench to follow the entire care pathway, on a well-defined cancer, nasopharyngeal carcinoma. The test bench is based on the longitudinal records of 405 patients from two cancer centers: 941 patient-step units spread over six successive phases, and 17,688 criteria specific to each patient, judged solely on the information available at the time of the decision.

Mistakes that change in nature

Out of 30 general and medical models tested, good diagnostic performance does not guarantee a good follow-up of care. The type of failure varies according to the phase: explicitly wrong decisions dominate anatomical staging (48.2%), while omissions prevail during the initial assessment (33.4%) and follow-up after treatment (30.4%). Models adapted to medicine omit more than general practitioners.

The wording of the questions also has an impact: open requests degrade the quality of decisions for all models, while detailed instructions, like a checklist, improve it, but at the cost of less fidelity to the data in the file and less compliance with safety rules. The authors conclude that static diagnostic benchmarks are not enough to guarantee the reliability of a clinical assistant over time.