This post is an informal preliminary writeup of a project that I've been working on with friends and collaborators. Some of the theory was developed jointly with Zohar Ringel, and we hope to write a more formal paper on it this year. Experiments are joint with Lucas Teixeira (and also Claude). This work is part of the research agenda we have been running with Lucas Teixeira and Lauren Greenspan at my organization Principles of Intelligence (formerly PIBBSS), and Lauren helped in writing this post.

Introduction

Pre-introduction

I have had trouble writing an introduction to this post. It combines three aspects of interpretability that I like and have thought about in the last two years:

Questions about computation in superposition
The mean field approach to understanding neural nets. This is a way of describing neural nets as multi-particle statistical theories, and has been applied in contexts of both Bayesian and SGD learning that has had significant success in the last few years. For example the only currently known exact theoretical prediction for the neuron distribution and grokking phase transition in a modular addition instance is obtained via this theory.
This connects to the agenda we are running at [...]

---

Outline:

(00:43) Introduction

(00:46) Pre-introduction

(04:38) The task

(07:11) Results

(09:20) The physics of frustration

(11:34) Tempering and quenched disorder in ML and interpretability

(14:43) Frustration in physics

(23:16) Continuous fluctuations from frozen structure

(27:25) Miscellanea: pretty pictures and future questions

The original text contained 8 footnotes which were omitted from this narration.

“A tale of three theories: sparsity, frustration, and statistical field theory” by Dmitry Vaintrob

LessWrong (30+ Karma)