In this post I'll discuss an apparent limitation of sparse autoencoders (SAEs) in their current formulation as they are applied to discovering the latent features within AI models such as transformer-based LLMs. In brief, I'll cover the following:

I'll argue that the L1 regularisation used to promote sparsity when training SAEs may cause neurons in the sparse layer to learn to represent common combinations of features rather than the individual features that we want them to discover
As well as making it more difficult to understand what the actual latent features are, I'll also argue that this limitation may result in some less common latent features not being discovered at all, not even within combinations
I'll then explain why I think that the phenomenon of feature splitting observed in Anthropic's SAE paper appears to demonstrate that this limitation does indeed have a large impact on the features discovered [...]

---

Outline:

(01:07) Rough definition of true features

(02:17) Why SAEs are incentivised to discover combinations of features rather than individual features

(08:33) Relation to feature splitting

(14:57) Proposed solution

---

First published:
February 22nd, 2024

Source:
https://www.lesswrong.com/posts/QoR8noAB3Mp2KBA4B/do-sparse-autoencoders-find-true-features

---

Narrated by TYPE III AUDIO.

“Do sparse autoencoders find ‘true features’?” by Demian Till

LessWrong (30+ Karma)