Yesterday Adam Shai put up a cool post which… well, take a look at the visual:

Yup, it sure looks like that fractal is very noisily embedded in the residual activations of a neural net trained on a toy problem. Linearly embedded, no less.

I (John) initially misunderstood what was going on in that post, but some back-and-forth with Adam convinced me that it really is as cool as that visual makes it look, and arguably even cooler. So David and I wrote up this post / some code, partly as an explainer for why on earth that fractal would show up, and partly as an explainer for the possibilities this work potentially opens up for interpretability.

One sentence summary: when tracking the hidden state of a hidden Markov model, a Bayesian's beliefs follow a chaos game (with the observations randomly selecting the update at each time), so [...]

---

Outline:

(01:11) Background: Fractals and Symmetry

(04:02) Background: Chaos Games

(05:58) Bayesian Belief States For A Hidden Markov Model

(09:35) Why Would That Show Up In A Neural Net?

(10:13) Why This Sort Of Thing Might Be A Pretty Big Deal

---

First published:
April 18th, 2024

Source:
https://www.lesswrong.com/posts/mBw7nc4ipdyeeEpWs/why-would-belief-states-have-a-fractal-structure-and-why

---

Narrated by TYPE III AUDIO.

“Why Would Belief-States Have A Fractal Structure, And Why Would That Matter For Interpretability? An Explainer” by johnswentworth, David Lorell

LessWrong (30+ Karma)

“Why Would Belief-States Have A Fractal Structure, And Why Would That Matter For Interpretability? An Explainer” by johnswentworth, David Lorell