Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

This is an interim report sharing preliminary results that we are currently building on. We hope this update will be useful to related research occurring in parallel.

Executive Summary

We train SAEs on base / chat model pairs and find that SAEs trained on the base model transfer surprisingly well to reconstructing chat activations (and vice versa) on Mistral-7B and Qwen 1.5 0.5B.
- We also find that they don’t transfer on Gemma v1 2B, and are generally bad at reconstructing <1% of unusually high norm activations (e.g. BOS tokens) from the opposite model.
We fine-tune our base Mistral-7B SAE (on 5 million chat activations) to cheaply obtain an SAE with competitive sparsity and reconstruction fidelity to a chat SAE trained from scratch (on 800M tokens).
We open source base, chat, and fine-tuned SAEs (plus wandb runs) [...]

---

Outline:

(00:22) Executive Summary

(01:46) Introduction

(04:14) Investigating SAE Transfer between base and chat models

(07:51) Identifying failures: Outlier norm activations and Gemma v1 2B

(10:24) Investigating SAE Transfer on Instruction Formatted Data

(12:48) Fine-tuning Base SAEs for Chat Models

(15:31) Conclusion

(16:20) Limitations

(17:57) Citing this work

(18:11) Author contributions Statement

“SAEs (usually) Transfer Between Base and Chat Models” by Connor Kissane, robertzk, Arthur Conmy, Neel Nanda

LessWrong (30+ Karma)