Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.This is a link post.

Abstract

Scalable oversight protocols aim to enable humans to accurately supervise superhuman AI. In this paper we study debate, where two AI's compete to convince a human judge; consultancy, where a single AI tries to convince a human judge that asks questions; and compare to a baseline of direct question-answering, where the human judge just answers outright without the AI. We use large language models (LLMs) as both AI agents and as stand-ins for human judges, taking the judge models to be weaker than agent models. We benchmark on a diverse range of asymmetries between judges and agents, extending previous work on a single extractive QA task with information asymmetry, to also include mathematics, coding, logic and multimodal reasoning asymmetries. We find that debate outperforms consultancy across all tasks when the consultant is [...]

---

Outline:

(00:14) Abstract

(01:50) Twitter thread

(01:53) Setup

(03:56) Assigned-role results

(05:32) Open-role results

(07:49) Persuasiveness in Debate

(10:31) Summary

(11:33) Limitations

(12:56) Future work

(14:58) Authors

“On scalable oversight with weak LLMs judging strong LLMs” by zac_kenton, Noah Siegel, janos, Jonah Brown-Cohen, Samuel Albanie, David Lindner, Rohin Shah

LessWrong (30+ Karma)