The search for extraterrestrial life: a Breakthrough Listen neural network detector misclassifies signals and mistakes interference for potential candidates

Tested on 15,000 signals injected into real data from the Green Bank telescope, a CNN detector sees its recall plummet as the signal becomes stronger and fails to recognise the patterns that would allow it to filter out interference.

To sort through the immense volume of data from the search for radio technosignatures, the teams rely on machine learning detectors, which narrow down the list of candidates before any human examination. The problem is that these detectors are validated on data similar to that of their training, whereas a real extraterrestrial signal would have an unknown shape and terrestrial interference is of an unlimited variety.

An eye-opening stress test

The author trained a convolutional array detector from a competition once, froze it, and then subjected it to 15,000 injections covering ten morphologies and five intensities, inserted into real bottoms registered at Green Bank. The recall did not follow the expected order, and it even dropped as the signals became brighter, including for the narrow signal type on which the model had been trained (from 0.590 to 0.090). One form of interference achieved a false positive rate of 0.627, comparable to the detector's basic recall on true signals (0.5525).

The explanation lies in the architecture: the detector only reads observations pointed at the target star (ON) and ignores lagged observations (OFF), which are precisely the only ones that can distinguish a celestial source from a local interference. It therefore does not know if a signal comes from the star, only if it looks like what it has already seen.