What would it take to get assurance that we have actually explained a neuron? Certainly, it would be a good sign if neuron activations looked like they had a consistent pattern, and a bad sign if they looked like a random jumble of unrelated concepts.

Nick Cammarata's recent post makes an important and true observation: in the past, people have been too quick to write off neurons as polysemantic, but if you look harder, there is actually often a simple pattern behind the activating examples. This historically caused people to overestimate how much polysemanticity there is. This is a really good point and we should update on it. But it's nowhere near a sufficient argument that we understand most of what neurons are doing, and these results do not show that interpretability is solved, or even that it is vastly easier than previously anticipated.

What does it take to explain a neuron?

It's unclear whether there even is a simple pattern that explains every neuron. But even if there is a simple pattern that matches the activating examples of a neuron, that still doesn’t mean that the explanation is correct. Suppose we look at the top activating examples of [...]

---

Outline:

(00:59) What does it take to explain a neuron?

(03:15) Simulation scores tell us whether we explained a neuron

(05:41) Beyond understanding neurons

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
October 17th, 2026

Source:
https://www.lesswrong.com/posts/Dkh7pXKjoNWkkxQDz/neural-networks-are-not-already-clean

---

Narrated by TYPE III AUDIO.

“Neural networks are not already clean” by leogao

LessWrong (30+ Karma)

“Neural networks are not already clean” by leogao