Brain MRI: a simple axis reversal causes an anomaly detection model’s performance to drop from 0.873 to 0.583

The MIRTO protocol reruns each comparison across 15,552 evaluation chains to demonstrate the extent to which method rankings depend on choices that are rarely published.

Unsupervised methods that detect lesions in brain MRIs are generally classified by a single score. However, this score depends on choices that are rarely documented: alignment of the anomaly maps with the reference, data used to set the threshold, budget for false positives, metric, definition of a lesion. The MIRTO protocol, proposed by two researchers, makes these choices explicit and measures their effect by replaying each comparison on 15,552 defensible evaluation chains.

Gaps that evaporate

Applied to four methods trained on the same healthy brains and tested on 312 patients from the BraTS 2020 database, the protocol reveals that a simple shift in the order of the axes between the stored maps and the reference caused the AUROC to fall to the voxel of a diffusion model from 0.873 to 0.583, with almost nothing changing its score per section. A significant advantage in Dice coefficient with thresholds set on validation, disappears when comparing methods with a truly equal charge of false positives.

While the method explains most of the variance for most metrics, it explains only 14% for lesion sensitivity, which is dominated by the way a detected lesion is defined. The authors specify that their conclusions remain exploratory, as the same cohort was used to design the protocol.