Having an LLM identify the clinical criteria in a message, rather than asking it to provide a direct score, makes the assessment more verifiable, but not more reliable, and fails to detect most severe cases.
To estimate the severity of depression from a social media post, a large language model can either directly give a level, or mark the clinical criteria present in the text and let a simple calculation convert their number into a label. The second approach has an obvious advantage: a clinician can check each criterion selected. A study published on arXiv compares the two on two Reddit corpora, with three models (of 9 billion parameters at the border) and two questionnaires, the PHQ-9 and the BDI-II.
More auditable, not more efficient
For the two state-of-the-art models, the extraction of criteria only goes beyond step-by-step reasoning on a single corpus, and only if its thresholds are adjusted on annotated data; the difference is never significant. With thresholds set in advance from the PHQ-9, no gain appears. The small model, on the other hand, classifies most messages from depression communities as severe, regardless of the instruction.
Importantly, better overall agreement does not guarantee better identification of severe cases: extraction according to PHQ-9 misses the majority of severe messages, and each step towards more structure increases these omissions in almost all comparisons. A simple model based on textual characteristics such as word count is on par with state-of-the-art models.
Read the full article on the original source
Go to the source →