This part-report / part-proposal describes ongoing research, but I'd like to share early results for feedback. I am especially interested in any comment finding mistakes or trivial explanations for these results. I will work on this proposal with a LASR Labs team over the next 3 months. If you are working (or want to work) on something similar I would love to chat!
Experiments and write-up by Stefan, with substantial inspiration and advice from Jake (who doesn’t necessarily endorse every sloppy statement I write). Work produced at Apollo Research.

TL,DR: Toy models of how neural networks compute new features in superposition seem to imply that neural networks that utilize superposition require some form of error correction to avoid interference spiraling out of control. This means small variations along a feature direction shouldn't affect model outputs, which I can test:

Activation plateaus: Real activations should be resistant to small [...]

---

Outline:

(02:22) Core results and discussion

(06:21) Proposal: Connecting SAEs to model behaviour

(09:19) Conclusion

(11:35) Appendix

(11:38) Methodology

(18:17) Detailed results

(18:21) 1. Activation Plateaus

(20:05) 2. Sensitive directions

(22:02) 3. Local optima in sensitivity

The original text contained 3 footnotes which were omitted from this narration.

The original text contained 7 images which were described by AI.

---

First published:
July 5th, 2024

Source:
https://www.lesswrong.com/posts/LajDyGyiyX8DNNsuF/interim-research-report-activation-plateaus-and-sensitive-1

---

Narrated by TYPE III AUDIO.

“[Interim research report] Activation plateaus & sensitive directions in GPT2” by StefanHex, jake_mendel

LessWrong (30+ Karma)