This work was produced as part of the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort, with support from Neel Nanda and Arthur Conmy. Joseph Bloom is funded by the LTFF, Manifund Regranting Program, donors and LightSpeed Grants. This post makes extensive use of Neuronpedia, a platform for interpretability focusing on accelerating interpretability researchers working with SAEs.

Links: SAEs on HuggingFace, Analysis Code

Executive Summary

This is an informal post sharing statistical methods which can be used to quickly / cheaply better understand Sparse Autoencoder (SAE) features.

Firstly, we use statistics (standard deviation, skewness and kurtosis) of the logit weight distributions of features (WuWdec[feature]) to characterize classes of features, showing that many features can be understood as promoting / suppressing interpretable classes of tokens.
- We propose 3 different kinds of features, analogous to previously characterized “universal neurons”:
  - Partition Features, which (somewhat) promote half [...]

---

Outline:

(00:39) Executive Summary

(05:38) Characterizing Features via the Logit Weight Distribution

(09:57) Token Set Enrichment Analysis

(10:26) What is Token Set Enrichment Analysis?

(10:46) Method Steps

(12:18) Case Studies

(24:01) Discussion

(24:04) Limitations

(25:34) Future Work

(29:36) Appendix

(29:39) Thanks

(30:08) How to Cite

(30:14) Glossary

(31:02) Prior Work

(32:37) Token Set Enrichment Analysis: Inspiration and Technical Details

(32:45) Inspiration

(34:16) Technical Details

(35:12) Results by Layer

---

First published:
March 11th, 2024

Source:
https://www.lesswrong.com/posts/qykrYY6rXXM7EEs8Q/understanding-sae-features-with-the-logit-lens

---

Narrated by TYPE III AUDIO.

“Understanding SAE Features with the Logit Lens” by Joseph Bloom, hijohnnylin

LessWrong (30+ Karma)

“Understanding SAE Features with the Logit Lens” by Joseph Bloom, hijohnnylin