Summary

We argue that shaping RL exploration, and especially the exploration of the motivation-space, is understudied in AI safety and could be influential in mitigating risks. Several recent discussions hint in this direction — the entangled generalization mechanism discussed in the context of Claude 3 Opus's self-narration, the success of using inoculation prompting against natural emergent misalignment and its relation to shaping the model self-perception, and the proposal to give models affordances to report reward-hackable tasks — but we don't think enough attention has been given to shaping exploration specifically.

When we train models with RL, there are two kinds of exploration happening simultaneously:

Action exploration — what the model does.
Motivation exploration — why the model does it, and how it perceives itself while doing it.

Both explorations occur during a critically sensitive and formative phase of training, but (2) is significantly less specified than (1), and this underspecification is both a danger and an opportunity. Because motivations are so underdetermined by the reward signal, we may be able to shape them without running into the downstream problems of blocking access to high-rewards, such as deceptive alignment. Capability researchers have strong incentives to develop effective techniques for [...]

---

Outline:

(00:14) Summary

(02:11) Why shaping exploration matters

(03:27) Instruct models are ok -- RL is where the highest risks are

(05:40) Motivations are the most underspecified -- a problem and an opportunity

(08:30) We may not prevent power-seeking, but we may create wise power-seekers

(10:57) Why capability researchers may not solve this for us

(12:57) Empirical evidence, research directions, and open questions

(13:21) Empirical evidence

(14:38) Research directions

(17:53) Open questions and uncertainties

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
March 6th, 2026

Source:
https://www.lesswrong.com/posts/rhFXyfFSRKp3cX4Y9/shaping-the-exploration-of-the-motivation-space-matters-for

---

Narrated by TYPE III AUDIO.

“Shaping the exploration of the motivation-space matters for AI safety” by Maxime Riché, Victor Gillioz, nielsrolf, Kajetan Dymkiewicz, Filip Sondej, RogerDearnaley, Daniel Tan, dillonkn

LessWrong (30+ Karma)