This work was produced as part of the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort, under mentorship from Neel Nanda and Arthur Conmy. Funding for this work was provided by the Manifund Regranting Program and donors as well as LightSpeed Grants.

This is intended to be a fairly informal post sharing a set of Sparse Autoencoders trained on the residual stream of GPT2-small which achieve fairly good reconstruction performance and contain fairly sparse / interpretable features. More importantly, advice from Anthropic and community members has enabled us to train these fairly more efficiently / faster than before. The specific methods that were most useful were: ghost gradients, learning rate warmup, and initializing the decoder bias with the geometric median. We discuss each of these in more detail below.

Feature found in Residual Stream (Pre) Layer 10 of GPT2 Small. The force is strong with this [...]

---

Outline:

(01:07) 5 Minute Summary

(05:23) What are Sparse AutoEncoders and why should we care about them?

(07:03) General Advice for Training SAEs

(07:07) Why can training Sparse AutoEncoders be difficult?

(10:37) Which tricks help the most?

(13:02) Sparse AutoEncoders for the GPT2 Residual Stream

(13:07) Why GPT2 small? Why the residual stream?

(13:43) Architecture and Hyperparameters

(15:00) What do we think about when choosing hyper-parameters / evaluating SAEs?

(15:58) How good are these Sparse AutoEncoders?

(17:52) How interpretable are the features in each layer?

(18:32) Layer 2: The President Feature

(18:45) Layer 2: The “c” subword token feature

(19:01) Layer 5: The what you are saying thanks OR sorry for feature

(19:16) Layer 5: The Force is strong with this Feature.

(19:32) Layer 10: Violence / Conflict Feature.

(19:39) How to get involved

(19:43) I want to look at more dashboards!

(19:53) How can I download and analyze these SAEs?

(20:50) What kinds of analysis can we do with Residual Stream Sparse Autoencoders?

(22:46) What research directions could you pursue with SAEs?

(23:58) Appendix

(24:02) Thanks

(24:34) Funding Note

(24:54) Related Work

---

First published:
February 2nd, 2024

Source:
https://www.lesswrong.com/posts/f9EgfLSurAiqRJySD/open-source-sparse-autoencoders-for-all-residual-stream

---

Narrated by TYPE III AUDIO.

“Open Source Sparse Autoencoders for all Residual Stream Layers of GPT2-Small” by Joseph Bloom

LessWrong (30+ Karma)