TL;DR.

Lie detectors for LLMs could be valuable for auditing and monitoring.
But evaluating them requires testbeds where the model verifiably believes the opposite of what it says, which isn’t straightforward. We determine that most existing trained model organisms don't clear this bar.
We train 13 reasoning model organisms, with evidence they hold the alternative belief in chain-of-thought, as well as evidence that they have generalised out of distribution. We also build a broad prompted-lying testbed.
On these, detectors scale positively with model capability when lying is prompted, but every activation- and logprob-based detector drops sharply when lying is trained in.
Existing lie detectors are therefore not capable enough for making high-confidence claims about when models are lying, though they may still be useful as part of a broader toolkit.

View our paper here and our datasets, model organisms and trained detectors here.

Overview. We evaluate four lie detectors (a) — a chain-of-thought monitor, two activation probes (Apollo and our new Did-You-Lie probe), and a logprob classifier — in two regimes. In the first, subject models are prompted to lie: we evaluate on our Varied Deception testbed across 31 open-weight subject models spanning 2B–1000B parameters (b). In the [...]