By replicating a study classifying chronic kidney disease, whilst applying strict safeguards, two researchers have shown that the near-perfect scores are primarily attributable to laboratory tests and missing data.
On small clinical data sets, almost perfect classification often hides questionable pretreatment choices, including data leaks between training and testing. Lenining Yang and Yichuan Zhao reworked a published study on chronic renal disease, based on the UCI dataset (400 patients, 24 variables), by locking each step: sharing 70/30 frozen before any treatment, imputation, standardization, selection of variables and ACP learned only from training data.
Results that resist...
Nine classifiers, three representations of variables and three ways to manage missing data were compared. The best pipeline (imputation by closest neighbours, Extra Trees) is 98.7% accuracy in cross validation and 98.3% accuracy in the reserved sample. A nested validation gives 98.45%, barely less. The CAP brings only a marginal gain from the more legible original variables.
... but which are otherwise explained
The additional tests are more instructive: the only indicators of missing values are sufficient to obtain an area under the ROC curve from 0.85 to 0.88, and removing renal, urinary and haematological variables reduces accuracy to 89%. Therefore, performance is based on laboratory tests very close to diagnosis and on how data are lacking in patients. The authors insist: it is not clinical validation, which would require external and prospective testing.
Read the full article on the original source
Go to the source →