Radiotherapy: Planning IA officers have more than 92% good decisions but let most of the impossible goals pass

The OpenGray test bench shows that LLM-based agents, well rated overall, report only a quarter to half of the anatomical contradictions they should raise.

Radiotherapy planners based on large language models decide themselves to submit a plan or request a rereading, but they are judged primarily on the dose delivered. Researchers propose OpenGray, a test bench built on the OpenKBP-Opt head and neck dataset, with tasks whose right decision is known in advance: demonstrated incompatible dose targets, or on the contrary reference plan respecting all constraints.

Four systems were tested on 30 patients, from a simple rule script to a complete agent, with a commercial LLM and a local model (Ministral 3 8B). With the business model, the overall accuracy of decisions is 95.4% and 92.1%, but systems only date back 50% and 25% of the contradictions related to organ overlap, which require anatomy calculations. In addition, 24 per cent of the plans submitted in feasible cases lacked an imperative target that the reference plan met.

Provide a stop button

The analysis of SAGE's newspapers, an agent used in stereotactic radiosurgery, shows an agent who identifies an impossible objective for each response without being able to stop, for lack of intended action. The authors recommend evaluating these agents on their decisions and explicitly giving them the opportunity to suspend and request a rereading.