Summary

Our CHIVE pipeline produced thousands of unexpected behaviors with explanations that are grounded in counterfactual prompts (see Figure 1 for an example). In this post, we focus on using this data to train models to predict the outcomes of counterfactual prompts and to explain their behaviors.

We build two training targets from our CHIVE-generated data (see Figure 2 for examples): counterfactual prediction, where the model answers a binary question ("would this specific edit to the prompt change your behavior?"), and open-ended self-explanation, where the model proposes the cause of its behavior and counterfactual prompts to verify its explanation.

We find three results:

1. Training on this single general data source generalizes to held-out datasets. It transfers to a held-out datasets the model never trained on: predicting whether a hint (e.g. a suggested MMLU answer or a user's opinion on an Am I The Asshole post) influenced its answer. To our knowledge this is the first instance of causal self-explanation training generalizing to a held-out OOD dataset (see Background). Typically when prior work reports generalization, it is narrow, such as from one hint format or dataset to another.

2. The counterfactual prediction training target substantially outperforms the open-ended one [...]

---

Outline:

(00:14) Summary

(02:49) en-US-AvaMultilingualNeural__ Diagram comparing prompts explaining Gemma's randomNum range error via parameter renaming.

(03:20) Background

(06:23) Setup

(07:45) Models and investigation setting

(09:02) Training targets

(10:42) Results

(10:45) Counterfactual prediction

(12:30) Open-ended self-explanation

(14:54) Is the self-explanation model introspecting?

(17:17) Discussion

---

First published:
September 4th, 2026

Source:
https://www.lesswrong.com/posts/YyAMz52wDxnLhwvWL/training-models-to-predict-and-explain-their-in-the-wild

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Diagram comparing prompts explaining Gemma's randomNum range error via parameter renaming.

Diagram comparing counterfactual claim and investigation explanation for randomNum code.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Training Models to Predict and Explain Their In-the-Wild Behavior” by Adam Karvonen, Subhash Kantamneni, Euan Ong, Sam Marks

LessWrong (30+ Karma)

“Training Models to Predict and Explain Their In-the-Wild Behavior” by Adam Karvonen, Subhash Kantamneni, Euan Ong, Sam Marks