UCLA researchers tested their model on 16 diseases and over 2 million patients in two health systems never seen in training, setting themselves at the realistic alert threshold for screening.
A UCLA team presents TRACER, a pre-trained decoder-type foundation model on the electronic medical records of more than 4.5 million patients from four health systems. Its role: to estimate, based on a patient's history, the risk that a disease will be diagnosed in one, two or five years. The authors evaluated it on 16 pathologies and more than 2 million patients from two other health systems, voluntarily excluded from the development phase.
Judging at the threshold of a real screening
Rather than sticking to the usual global metrics, the study measures performance with very few false alarms, as would be required by a screening program. At 95% specificity, patients reported by TRACER were up to eight times more likely to receive the diagnosis within a year than an unselected population, and the model retained its discriminating power in external settings.
This more demanding point of view also reveals what the overall scores mask: the ability to detect and the burden of false alarms vary markedly according to the disease and the hospital, even when discrimination appears to be stable. The authors argue that these models should be systematically validated in health systems that were not used in their design, at the alert level corresponding to the intended clinical use.
Read the full article on the original source
Go to the source →