TL;DR: We use a suite of testbed settings where models lie—i.e. generate statements they believe to be false—to evaluate honesty and lie detection techniques. The best techniques we studied involved fine-tuning on generic anti-deception data and using prompts that encourage honesty.

Read the full Anthropic Alignment Science blog post and the X thread.

Introduction:

Suppose we had a “truth serum for AIs”: a technique that reliably transforms a language model $M$ into an honest model $M_H$ that generates text which is truthful to the best of its own knowledge. How useful would this discovery be for AI safety?

We believe it would be a major boon. Most obviously, we could deploy $M_H$ in place of $M$. Or, if our “truth serum” caused side-effects that limited $M_H$'s commercial value (like capabilities degradation or refusal to engage in harmless fictional roleplay), $M_H$ could still be used by AI developers as a tool for ensuring $M$'s safety. For example, we could use $M_H$ to audit $M$ for alignment pre-deployment. More ambitiously (and speculatively), while training $M$, we could leverage $M_H$ for oversight by incorporating $M_H$'s honest assessment when assigning rewards. Generally, we could hope to use $M_H$ to detect [...]

---

First published:
November 25th, 2025

Source:
https://www.lesswrong.com/posts/9f7JmoaMfwymgsW9S/evaluating-honesty-and-lie-detection-techniques-on-a-diverse

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Diagram showing five testbeds of model dishonesty: Harm Pressure, Password Lock, MASK, SSC, and Auditing MO.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Evaluating honesty and lie detection techniques on a diverse suite of dishonest models” by Sam Marks, Johannes Treutlein, evhub, Fabien Roger

LessWrong (30+ Karma)

“Evaluating honesty and lie detection techniques on a diverse suite of dishonest models” by Sam Marks, Johannes Treutlein, evhub, Fabien Roger