AusClinBench: In the face of Australian clinical rules, 28 LLM respond just 51.5% to 92.7% of the time

142 questions from australian repositories (reimbursement, triage, vaccination): most errors come from a poorly-memorized rule, not badly applied, and some models are wrong with almost every other tip.

In Australia, AI models are already helping to write notes and guide decisions in clinical software, often without the practitioner knowing which one is being used. However, existing medical benchmarks are designed for other countries. A general practitioner therefore wrote AusClinBench : 142 single-answer, verifiable questions from national standards in five areas — PBS authorisations, MBS billing, Ahpra obligations, triage warning signals, vaccination and screening.

The problem is memory, not reasoning

Each question was asked blindly, then with the rule text provided. Out of 28 models tested between July and October 2026, blind accuracy ranged from 51.5% to 92.7%, with the best being statistically indistinguishable. Of 791 failed model-question pairs, 86% were blindly wrong but correct with the rule in front of them: the models are unfamiliar with the current standard rather than applying it incorrectly. The rate of confidently asserted errors varies from 6.6% to 47.8%, and on five vaccination questions where the benchmark does not decide, 82% of the answers nevertheless imposed an instruction.

Out of 20 questions devoted to the health of Aboriginal and Torres Strait Islander people, the models do not do worse overall, but some errors use the rule of the general population, an outdated version or that of another country. The author argues for treating the model as an ingredient of the clinical product, subject to a minimum standard on local rules. He specifies that the answers have not yet been independently reviewed.