Synthetic Hospital: 1,268 patients created from scratch, and doctors who only spot them 53 per cent of the time

A set of entirely synthetic longitudinal medical records, containing no sensitive data, from which ten models are able to retrieve the information but fail to reconstruct it.

A team at Carnegie Mellon publishes Synthetic Hospital, a set of longitudinal electronic medical records entirely fabricated: 1,268 patients followed over time and 5,602 consultations, built from public medical teaching materials, therefore without any protected health data. The objective is to remove two barriers to the field at once — the impossibility of sharing real records, and the lack of reliable reference truth.

A truth of reference that does not depend on what has been noted

Each diagnosis, observation, result and narrative statement is linked by a typed knowledge graph to concepts standardized in ICD-10-CM, SNOMED CT, and LOINC, with traceability back to the source document. The reference therefore does not depend on what a clinician has thought to document. In blind reading, doctors distinguished these synthetic records from real MIMIC-IV records in only 53% of cases, which is only slightly better than chance.

None of the ten models evaluated masters the game. The failure profile is consistent: they correctly find information present in a file, but collapse as soon as it has to be assembled into a diagnosis or an usable summary. When the elements are scattered over several consultations, the agents who go to look for the information on their own are ahead of the models questioned in a single round by 0.14 to 0.34 F1 points.