NOEMA: identifying, in the brain activity measured by fMRI, the passage of speech that a person was listening to

Out of 2,000 candidate excerpts, the decoder ranks the correct one in its top 10 for more than 42 per cent of queries, by combining representations of text, audio and speech.

Reconstitute what a person perceives from his or her cerebral recordings is a good test of the richness of these signals. Natural speech lends itself to this, but functional MRI measures noise and spread over time, blurring the fine structure of the language. A team, including several members of the French research ecosystem around Bertrand Thirion, presents NOEMA, a temporal convolutive decoder.

Rather than treating each measure in isolation, NOEMA takes a window of consecutive measurements and aligns it with a representation of the heard segment that combines three complementary points of view: text, audio signal and speech-specific representations. The right segment can then be found directly from brain activity.

On two sets of natural listening data, Cephalonauts One and LeBel, the system places the correct extract in its first ten choices among 2,000 candidates for more than 42% of the unreleased queries, well above the reference decoders, and keeps its lead when adding thousands of lures missing from the training. Ablations show that both temporal modelling and the combination of representations contribute to the result.