Audio note: this article contains 88 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

This post presents a simple toy coherence theorem, and then uses it to address various common confusions about coherence arguments.

Setting

Deterministic MDP. That means at each time <span>_t_</span> there's a state <span>_S[t]_</span>[1], the agent/policy takes an action <span>_A[t]_</span> (which can depend on both time <span>_t_</span> and current state <span>_S[t]_</span>), and then the next state <span>_S[t+1]_</span> is fully determined by <span>_S[t]_</span> and <span>_A[t]_</span>. The current state and current action are sufficient to tell us the next state.

We will think about values over the state at some final time <span>_T_</span>. Note that often in MDPs there is an incremental reward each timestep in addition to a final reward at the end; in our setting there is zero [...]

---

Outline:

(00:25) Setting

(01:30) Theorem

(01:48) Proof

(07:51) Anti-Takeaways: Things Which Dont Generalize

(07:57) Determinism

(08:48) Takeaways: Things Which (I Expect) Do Generalize

(08:55) Coherence Is Nontrivial For Optimization At A Distance

(11:18) We Didnt Need Trades, Adversaries, Money-Pumping, Etc

(12:06) Approximation

(12:54) Coherence Is About Revealed Preferences

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
August 2nd, 2024

Source:
https://www.lesswrong.com/posts/DXxEp3QWzeiyPMM3y/a-simple-toy-coherence-theorem

---

Narrated by TYPE III AUDIO.

“A Simple Toy Coherence Theorem” by johnswentworth, David Lorell

LessWrong (30+ Karma)