This is a link post.

When you think you've found a circuit in a language model, how do you know if it does what you think it does? Typically, you ablate / resample the activations of the model in order to isolate the circuit. Then you measure if the model can still perform the task you're investigating.

We identify six ways in which ablation experiments often vary.[1][2]

How do these variations change the results of experiments that measure circuit faithfulness?

TL;DR

We study three different circuits from the literature and find that measurements of their faithfulness are highly dependent on details of the experimental methodology. The IOI and Docstring circuits in particular are much less faithful than reported when tested with a more precise methodology.
The correct circuit for a set of prompts is undefined. The type of ablation you use to isolate the circuit determines the task that [...]

---

Outline:

(00:43) TL;DR

(01:29) Case Studies

(01:41) Indirect Object Identification Circuit

(06:28) Docstring Circuit

(09:01) Sports Players Circuit

(10:15) Methodology Should Match the Circuit

(10:45) Optimal Circuits are Defined by Ablation Methodology

(13:40) AutoCircuit

The original text contained 7 footnotes which were omitted from this narration.

The original text contained 2 images which were described by AI.

“Transformer Circuit Faithfulness Metrics Are Not Robust” by Joseph Miller, bilalchughtai, William_S

LessWrong (30+ Karma)