Nick and Lily are co-first authors on this project. Lewis and Neel jointly supervised this project.

TL;DR

We use sparse autoencoders (SAEs) for four textual data analysis
tasks—data diffing, finding correlations, targeted clustering, and retrieval.
We care especially about gaining insights from language model data, such as LLM outputs and training data, as we believe it is an underexplored route for model understanding.
For instance, we find that Grok 4 is more careful than other frontier models to state its assumptions and explore nuanced interpretations—showing the kinds of insights data diffing can reveal by comparing model outputs.
Why SAEs? Think of features as "tags" of properties for each text.
- Their large dictionary of latents provides a large hypothesis space, enabling the discovery of novel insights (diffing and correlations).
- SAEs capture more than just semantic information, making them effective alternatives to traditional embeddings when we want to find [...]