This post was written as part of research done at MATS 9.0 under the mentorship of Richard Ngo. It's related to my previous post, but should be readable as a standalone.

Remark: I'm not yet familiar enough with the active inference literature to be sure that the issues I bring up haven't been addressed or discussed. If you think my characterisation of the state and flaws of the theory are missing something substantial, I'd love to know.

Introduction

In the theory of active inference, agents are described as having a set of internal states that interact with external states (the world) through a membrane of intermediate states, such as the senses. I'm currently exploring how agents are able to exhibit approximations of external reference that allow them to stay alive in the real world. They achieve this even though they only have access to the statistical proxy of their internals, which they could easily reward-hack without optimising the external states at all.

One of active inference's weaknesses is that it struggles to model agents' uncertainties about their own preferences. I here propose a potential explanation for why agents are conflicted about these preferences. This perspective posits agents' seeming inconsistency [...]

---

Outline:

(00:37) Introduction

(01:30) Internal reward-hacking

(02:13) Do agents just have good priors?

(03:48) Internal inconsistency as a tool for rationality

(05:26) Whats next: competition between preferences

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
January 17th, 2026

Source:
https://www.lesswrong.com/posts/H8uoAmbeqjD2PG2jm/irrationality-as-a-defense-mechanism-for-reward-hacking

---

Narrated by TYPE III AUDIO.

“Irrationality as a Defense Mechanism for Reward-hacking” by Ashe Vazquez Nunez

LessWrong (30+ Karma)

“Irrationality as a Defense Mechanism for Reward-hacking” by Ashe Vazquez Nunez