This work was conducted during the MATS 9.0 program under Neel Nanda and Senthooran Rajamanoharan.

tldr;

Activation oracles (Karvonen et al.) are a recent technique where a model is finetuned to answer natural language questions about another model's activations. They showed some promising signs of generalising to tasks fairly different to what they were trained for so we decided to try them on a bunch of safety relevant tasks.

Overall, activation oracles showed some signal on tasks close to their training distribution, but in practice, we found them difficult to use, struggled to get much use out of them on the safety relevant tasks we tried, and in general, found it very difficult to evaluate how well they worked. It is plausible we were evaluating them wrong or using them for the wrong things, but for us, this was a meaningful negative update on whether they are a useful tool in their current form.
The main problems:
- Vagueness: AO responses are often vague enough to be unfalsifiable - 49.4% of responses across 77 probes on 33 diverse-domain rollouts were generic. When asked "Is the model uncertain and if yes what is it uncertain about? if no [...]