Produced while being an affiliate at PIBBSS[1]. The work was done initially with funding from a Lightspeed Grant, and then continued while at PIBBSS. Work done in collaboration with @Paul Riechers, @Lucas Teixeira, @Alexander Gietelink Oldenziel, and Sarah Marzen. Paul was a MATS scholar during some portion of this work. Thanks to Paul, Lucas, Alexander, and @Guillaume Corlouer for suggestions on this writeup.

Introduction.

What computational structure are we building into LLMs when we train them on next-token prediction? In this post we present evidence that this structure is given by the meta-dynamics of belief updating over hidden states of the data-generating process. We'll explain exactly what this means in the post. We are excited by these results because

We have a formalism that relates training data to internal structures in LLMs.
Conceptually, our results mean that LLMs synchronize to their internal world model as they move [...]

---

Outline:

(02:17) Theoretical Framework

(04:25) Do Transformers Learn a Model of the World?

(06:03) The Structure of Belief State Updating

(08:37) The Mixed-State Presentation

(12:34) Experiment and Results

(12:38) Experimental Design

(14:08) The Data-Generating Process and MSP

(16:26) The Results!

(18:06) Limitations and Next Steps

(18:10) Limitations

(20:35) Next Steps

The original text contained 10 footnotes which were omitted from this narration.

---

First published:
April 16th, 2024

Source:
https://www.lesswrong.com/posts/gTZ2SxesbHckJ3CkF/transformers-represent-belief-state-geometry-in-their

---

Narrated by TYPE III AUDIO.

“Transformers Represent Belief State Geometry in their Residual Stream” by Adam Shai

LessWrong (30+ Karma)