TLDR: We want to describe a concrete and plausible story for how AI models could become schemers. We aim to base this story on what seems like a plausible continuation of the current paradigm. Future AI models will be asked to solve hard tasks. We expect that solving hard tasks requires some sort of goal-directed, self-guided, outcome-based, online learning procedure, which we call the “science loop”, where the AI makes incremental progress toward its high-level goal. We think this “science loop” encourages goal-directedness, instrumental reasoning, instrumental goals, beyond-episode goals, operational non-myopia, and indifference to stated preferences, which we jointly call “Consequentialism”. We then argue that consequentialist agents that are situationally aware are likely to become schemers (absent countermeasures) and sketch three concrete example scenarios. We are uncertain about how hard it is to stop such agents from scheming. We can both imagine worlds where preventing scheming is incredibly [...]

---

Outline:

(01:46) Introduction

(05:31) A plausible story for how scheming could arise

(05:36) We start with an improved version of today's systems

(08:04) The AI is put into a self-supervised “Science Loop”

(11:11) Hypothesis: The “science loop” will reward Consequentialism

(14:51) Goal-directedness

(16:03) Instrumental reasoning

(17:17) Instrumental goals

(18:46) Stable beyond-episode goals

(20:00) Operational non-myopia

(21:02) Indifference to others’ preferences

(22:26) Extra: Goal drift

(23:59) Hypothesis: Consequentialism will lead to scheming

(24:16) AI will have misaligned goals

(26:35) AI will seek power

(27:15) AI will use strategic deception to achieve its goals

(32:27) Concrete scenarios for catastrophic outcomes from this setup

(32:56) Abstract hard goal leads to takeover of the US government

(38:09) AI is tasked with “building safe ASI”

(42:51) Race between superpowers leads to loss of control

(46:05) We’re very uncertain how hard it is to address scheming

(49:24) Evidence for or against scheming to look out for

(52:29) Appendix A - discussing definitions of scheming

(54:23) Appendix B - comparing with other stories of scheming

---

First published:
November 19th, 2024

Source:
https://www.lesswrong.com/posts/QqYfxeogtatKotyEC/training-ai-agents-to-solve-hard-problems-could-lead-to

---

Narrated by TYPE III AUDIO.

“Training AI agents to solve hard problems could lead to Scheming” by Marius Hobbhahn, AlexMeinke

LessWrong (30+ Karma)