Predicting sepsis in intensive care: whilst performance is equivalent, two imputation pipelines do not flag the same patients

A study involving more than 40,000 patients shows that the choice of method for imputing missing data alters the list of patients identified, even when the AUROC remains the same.

In intensive care records, missing data is the norm. Before training a predictive model, these must therefore be ‘imputed’, i.e. filled in. This pre-processing is usually assessed using the reconstruction error or the overall AUROC, two metrics that do not indicate whether the same patients trigger an alert. A research team proposes a complementary measure: alert list concordance – the Jaccard similarity between patients placed in the top k per cent at risk by two pipelines, with confidence intervals calculated on a patient-by-patient basis.

Lists that only overlap by half

Tested on the PhysioNet/CinC 2019 dataset (40,336 patients, 71.9 per cent missing values) using nine imputation strategies and the same XGBoost model, the results speak for themselves: for 22 of the 28 pairs of methods, no significant difference in AUROC emerges after Bonferroni correction, but their alert lists overlap by only between 0.52 and 0.75. Imputation using the last known value stands out with better discrimination but the lowest overlap (0.40 to 0.45).

Deep imputation methods do not perform better than simple averaging in terms of discrimination, whilst taking 5 to 12 times longer to compute. The authors argue that pre-processing poses an underestimated risk to clinical reliability.