Summary: Superposition-based interpretations of neural network activation spaces are incomplete. The specific locations of feature vectors contain crucial structural information beyond superposition, as seen in circular arrangements of day-of-the-week features and in the rich structures. We don’t currently have good concepts for talking about this structure in feature geometry, but it is likely very important for model computation. An eventual understanding of feature geometry might look like a hodgepodge of case-specific explanations, or supplementing superposition with additional concepts, or plausibly an entirely new theory that supersedes superposition. To develop this understanding, it may be valuable to study toy models in depth and do theoretical or conceptual work in addition to studying frontier models.

Epistemic status: Decently confident that the ideas here are directionally correct. I’ve been thinking these thoughts for a while, and recently got round to writing them up at a high level. Lots of people (including [...]

---

Outline:

(01:26) What would it mean if we could fully understand an activation space through the lens of superposition?

(02:58) The placement of each feature vector in the activation space matters

(07:40) What types of theories could fill this gap in our understanding?

(07:59) Option 1: A hodge-podge of explanations

(08:43) Option 2: Supplementing superposition

(10:15) Option 3: Superseding superposition

(12:11) How could we discover the new theory?

(12:23) Approach 1: Investigating feature structure in big SAEs

(13:18) Approach 2: Directly look for interesting structure in LLM representations

(14:34) Approach 3: Carefully reverse engineering relevant toy models with ground truth access

(16:27) Approach 4: Theoretical work to unite experimental results and motivate new experiments

(17:54) Acknowledgements

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
June 24th, 2024

Source:
https://www.lesswrong.com/posts/MFBTjb2qf3ziWmzz6/sae-feature-geometry-is-outside-the-superposition-hypothesis

---

Narrated by TYPE III AUDIO.

“SAE feature geometry is outside the superposition hypothesis” by jake_mendel

LessWrong (30+ Karma)