TL;DR

Your base LLM has no examples of superintelligent AI in its training data. When you RL it into superintelligence, it will have to extrapolate to how a superintelligent Claude would behave. The LLM's extrapolation may not converge optimizing for what humanity would, on reflection, like to optimize, because these are different processes with different inductive biases.

Intro

I'm going to take the Persona Selection Model as being roughly true, for now. Even on its own terms, it will fail. If the Persona Selection Model is false, we die in a different way.

I'm going to present some specific arguments and secnarios, but the core of it is a somewhat abstract point: the Claude persona, although it currently behaves in a human-ish way, will not grow into a superintelligence in the same way that humans would. This means it will not grow into the same kind of superintelligence with the same values that human values would converge on. Since value is fragile, this is fatal for the future.

I don't think this depends on the specifics of Claude's training, nor how human values are instantiated, unless Claude's future training methods are specifically designed to work in the exact [...]

---

Outline:

(00:10) TL;DR

(00:36) Intro

(01:35) LLMs

(01:39) Persona Selection and Other Models

(03:06) Persona Theory As Alignment Plan

(04:21) Gears of Personas

(07:21) Complications

(08:20) Reasoning and Chain-of-thought

(10:00) Reinforcement Learning

(11:40) Humans

(11:43) Human Values

(12:00) TL;DR

(12:30) Goal-Models and Inductors

(13:49) These Are Not The Same

(16:34) Final Thoughts

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
March 1st, 2026

Source:
https://www.lesswrong.com/posts/fMgE3E54PdDcZhvm6/i-m-bearish-on-personas-for-asi-safety

---

Narrated by TYPE III AUDIO.

“I’m Bearish On Personas For ASI Safety” by J Bostock

LessWrong (30+ Karma)