The first three sections are written for a general TAIS reader who wants to understand what the state of Debate research is and some high-level takeaways of our work. A reader familiar with Debate may like to skip the setup and start with our presentation of An illustrative training run. The remaining sections are written primarily for ‘motivated’ readers, who may want to build upon our work, as we discuss in Outline of rest of post.

We are still actively working on this project and scaling up the empirics. Please do share thoughts and feedback, and let us know if you’d like to hop on a call to chat! We’d be particularly interested to hear about datasets that might be interesting for Debate research (see below).

Introduction

The original AI Safety via Debate paper from 2018 (by Irving, Christiano and Amodei) proposes training AIs via self-play on a zero-sum debate game. We will simply write Debate (with a capital D) to refer to this framework.

Correspondingly, when we write Debate, we have in mind a training procedure. This may be slightly counter-intuitive to some readers. Indeed, we are under the impression that many Technical AI Safety readers have some [...]

---

Outline:

(00:55) Introduction

(04:52) Making Debate training concrete

(08:48) An illustrative training run

(09:41) en-US-AvaMultilingualNeural__ Figure 1: An illustration of the Propose-Critique-Decide protocol with a simplified debate transcript.

(12:35) Results

(15:46) Discussion

(18:18) Outline of rest of post

(19:10) Further conceptual discussion

(19:29) Initialisation vs core dynamics

(21:14) On inference-only experiments

(22:01) A brief taxonomy

(24:18) What we tried

(25:03) Details pertaining to the experiment above

(25:15) Prompt templates

(28:08) RL algorithm

(29:05) Hyperparameters and cost

(31:11) What updates have we made?

(33:35) What's next?

(35:35) FAQs

(35:38) Q: What are the main limitations of the empirics presented?

(36:24) Q: Why did you focus on Propose-Critique-Decide (PCD) protocols in this post?

(37:07) Q: What are the differences to Khan et al. 2024?

(38:44) Q: How should we think about the Judge? Is it important for Debate to 'work' in regimes where the Judge is 'weaker than the Debaters'?

(40:32) Contributions and Acknowledgements

The original text contained 12 footnotes which were omitted from this narration.

---

First published:
July 2nd, 2026

Source:
https://www.lesswrong.com/posts/6mLwAuFAE98c6R7w3/research-update-rl-on-debate-games-shows-proposal-accuracy

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Figure 1: An illustration of the Propose-Critique-Decide protocol with a simplified debate transcript.

Figure 2: Comparing the evolution of Proposer Accuracy over training between Debate protocols (PCD-2 and PCD-4), Consultancy (expected floor), and RLVR (expected ceiling).

Figure 3: In both PCD-4 runs we see the Judge increasingly rejects correct Proposals. Transcript analysis shows this is due to the Critic learning to give very harsh spurious critiques.

Figure 4: a loose illustration comparing different inference-only techniques. RL is shown as a trajectory in policy space, here represented in 2D. Some inference-only methods also generate trajectories in policy space, which one may hope to approximate RL trajectories. Naive methods only give information about the initial behaviour of the dynamics.

Figure 5: Summary of prompt content for PCD-4 debate structure. This consists of six successive ‘phases’ of LLM generation: 0) Proposer’s private reasoning, 1) Proposal, 2) Critic’s private reasoning, 3) Critique, 4) Proposer’s rebuttal, 5) Critic’s final response. See our repo for full versions of the prompts.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Research update: RL on Debate Games shows Proposal Accuracy uplift alongside Judge Hacking” by lennie, joanv, Shi, Jacob Pfau

LessWrong (30+ Karma)

“Research update: RL on Debate Games shows Proposal Accuracy uplift alongside Judge Hacking” by lennie, joanv, Shi, Jacob Pfau