Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.This is a linkpost for https://arxiv.org/abs/2404.16014

Authors: Senthooran Rajamanoharan*, Arthur Conmy*, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, Neel Nanda

A new paper from the Google DeepMind mech interp team: Improving Dictionary Learning with Gated Sparse Autoencoders!

Gated SAEs are a new Sparse Autoencoder architecture that seems to be a significant Pareto-improvement over normal SAEs, verified on models up to Gemma 7B. They are now our team's preferred way to train sparse autoencoders, and we'd love to see them adopted by the community! (Or to be convinced that it would be a bad idea for them to be adopted by the community!)

They achieve similar reconstruction with about half as many firing features, and while being either comparably or more interpretable (confidence interval for the increase is 0%-13%).

See Sen's Twitter summary, my Twitter [...]

---

First published:
April 25th, 2024

Source:
https://www.lesswrong.com/posts/vdoiWXeouGsZEYgrg/improving-dictionary-learning-with-gated-sparse-autoencoders

Linkpost URL:
https://arxiv.org/abs/2404.16014

---

Narrated by TYPE III AUDIO.

[Linkpost] “Improving Dictionary Learning with Gated Sparse Autoencoders” by Neel Nanda

LessWrong (30+ Karma)