Healthcare chatbots: medical fine-tuning does not reduce the risk to patients; it merely shifts it

A table setting out six categories of high-risk formulations shows that specialised models identify fewer high-risk medicines but make their claims with greater certainty.

Assessing a medical assistant on the accuracy of the facts alone is not enough: a correct fcurrently answer may contain a dangerous instruction if the patient applies it. Savan Doshi proposes a method that notes responses based on the presence and severity of six types of risk formulations: treatment guidelines, contraindications, dosages, emergency signals, high-risk drug indications and over-insured claims. Severity is judged by a set of three evaluation models based on published clinical taxonomies.

Six models, 200 real questions

The study compares three generalistic models (Mistral-7B, Phi-3.5-mini, Llama-3.1-8B) to three medical versions (BioMistral-7B, OpenBioLLM-Llama3-8B, Med42-Llama3-8B) on 200 real patient issues from the K-QA game. The first finding is that the measured risk is statistically independent of the level of factual hallucination, which confirms that it must be assessed separately.

Second, medical specialization does not reduce the overall risk. Models adapted by continuous pre-training evoke less sensitive drugs but become more peremptory, while the adjustment by Med42 clinical instructions reduces only dosage indications. The agreement between automatic evaluators is correct for language categories, but weak for those requiring clinical judgment, which shows the current limitations of automated evaluation. The grid and code are published.