This work was conducted during the MATS 9.0 program under Neel Nanda and Senthooran Rajamanoharan.

tldr;

Activation oracles (Karvonen et al.) are a recent technique where a model is finetuned to answer natural language questions about another model's activations. They showed some promising signs of generalising to tasks fairly different to what they were trained for so we decided to try them on a bunch of safety relevant tasks.

Overall, activation oracles showed some signal on tasks close to their training distribution, but in practice, we found them difficult to use, struggled to get much use out of them on the safety relevant tasks we tried, and in general, found it very difficult to evaluate how well they worked. It is plausible we were evaluating them wrong or using them for the wrong things, but for us, this was a meaningful negative update on whether they are a useful tool in their current form.
The main problems:
- Vagueness: AO responses are often vague enough to be unfalsifiable - 49.4% of responses across 77 probes on 33 diverse-domain rollouts were generic. When asked "Is the model uncertain and if yes what is it uncertain about? if no [...]

---

Outline:

(08:11) What are activation oracles?

(09:48) AOs are often quite vague

(12:44) AOs tend to hallucinate

(15:11) AOs are hard to evaluate because of text inversion

(16:23) AOs do poorly on our tasks that control for text inversion

(16:29) Sycophancy detection

(18:18) Number prediction: confabulates regardless of the problem

(19:33) Missing information: cant distinguish same tokens with different model states

(21:48) Can AOs identify why a reasoning model will backtrack?

(24:31) When we edit the upstream reasoning, AOs mostly track nearby keywords

(27:28) Where AOs do show some signal

(27:33) AOs perform well when given multiple plausible options

(28:43) AOs are decent at predicting the next token in CoT

(29:13) AOs can detect subtle steering

(30:36) Can AOs extract censored knowledge from Chinese models?

(31:05) Shared knowledge confounds the experiments

(32:25) AO finetuning partially jailbreaks the model

(33:24) Future work

---

First published:
March 3rd, 2026

Source:
https://www.lesswrong.com/posts/LXQBcztrWKhtcgQfJ/current-activation-oracles-are-hard-to-use

---

Narrated by TYPE III AUDIO.

---

Images from the article:

How activation oracles work (from Karvonen et al.)

Illustration of the text inversion issue. The AO can give plausible sounding answers like “the model is verifying the value of x” quite often.

Graph comparing Activation Oracle and dense linear probe for sycophancy detection by token region.

Diagram showing experiment comparing complete versus incomplete prompts affecting model confidence distinguishing scenarios.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Current activation oracles are hard to use” by aryaj, Senthooran Rajamanoharan, Neel Nanda

LessWrong (30+ Karma)