“Neural networks are already clean”: the superposition did not explain the indecipherable neurons, argues a researcher at Goodfire

According to a lengthy essay published by Goodfire, ‘polysemantic’ neurons are primarily the result of architectural flaws that can be corrected. Modern networks are said to remain interpretable even with trillions of parameters.

An essay published on a website of Goodfire, a specialist in interpretability, calls into question an idea that has been structuring this field for several years: that of "superposition", according to which a network running out of neurons would fold unrelated concepts on top of each other. This hypothesis had led a large number of researchers to study not the network itself, but an "unfolded" version produced by a second model, such as a sparse autoencoder.

Ordinary flaws, not a hidden principle

The author, who worked with Chris Olah on the circuits of InceptionV1, claims that the founding examples of polysemanticity have been misinterpreted. In this model of vision, the last layers seemed incomprehensible because of architectural problems that were amplified by too weak analysis tools. With usual corrections, without changing the size of the model or its data, 99 to 100% of the neurons in the last layer become easy to interpret. Trained to sizes covering a factor of 300, the network remains readable: with little space, a neuron covers a large concept ("car"), with more, it becomes specialized. What looked like arbitrary folding would be mostly a form of recursive generalization.

The essay extends this observation to language models: most of their neurons would be understandable, even in models of several trillion parameters. The author presents his conclusions as a working hypothesis, but believes that mechanistic interpretability could once again study networks directly.