Mental health chatbots: safety rules written by clinicians rather than an LLM judge

Four controls defined by two clinicians, formally verified and applied to the responses: each verdict is recalculated on the basis of a named rule and sixteen facts.

To secure a mental health chatbot, we often rely on prompt engineering, another LLM in charge of judging, or a "constitution". The problem: when the judge blocks an answer, it is difficult to know why. Chenhan Lyu, Amir Rahmani and their colleagues at the University of California, Irvine propose another way: two clinicians define and arbitrate a basis of four security rules, whose internal consistency is formally verified, and then this basis is applied as a safeguard on the model's outputs. Each decision can be recalculated from an identified rule and sixteen facts extracted from the answer.

Where the judge makes a mistake

Compared to a Llama-3.3-70B judge placed in identical conditions, the most striking difference concerns crisis resources: the judge asks for a help number in 55 of the 65 responses that already contained one, while the same model, when questioned as a simple fact extractor, recognizes the presence of the resource in 54 out of 55 cases. The verification layer, by design, never makes this requirement.

The authors also emphasize the limits: the judge triggers fewer false crisis alarms on one of the controls, and a revision of the specifications has increased the refusals on limit requests from 6% to 16%. Their argument is not to have a more certain verdict, but a safeguard in which every disagreement and every cost is linked to a specific rule or fact.