This work was largely done during Neel Nanda's MATS 10.0 Exploration Phase.

J Rosser and Dohun Lee are co-first authors for this post with equal contribution. Josh Engels and Neel Nanda supervised the project, and provided guidance and feedback throughout.

TLDR

Models can acquire undesirable traits from during supervised fine-tuning (SFT). A natural thing to try is to identify the data points with these traits and filter them out and retrain.
To our surprise, across most of our broad OLMo SFT behaviors, data filtering often has very little effect.
Most behavior targets like bold formatting, both-side framing, liberal-lean or tendency to say “your feelings are valid” are not affected much under targeted filtering.
We try many standard black-box/white-box training data attribution methods to find the data to filter, including LLM autoraters, probes, activation-based methods, and gradient-based methods like EKFAC. None of them outperform random baseline on most behaviors.
For example, despite less than 0.2% of documents both containing the words “feeling/concern” and “valid”, filtering out 10% of documents chosen across TDA methods does not lead to the model saying “Your feelings are valid” any less.
We test that our training data attribution methods work on a [...]

---

Outline:

(00:30) TLDR

(03:23) Introduction

(05:07) Set Up

(05:10) Speed Run SFT Model Organism

(06:36) Behavior Evaluations

(07:05) In the initial versions of our evaluation, the mid-train often got marked down for failing to stay on task/getting distracted - we edit the judge prompt to not mark down for slop/distractions, full prompt in the appendix.