Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

Thanks to Clément Dumas, Nikola Jurković, Nora Belrose, Arthur Conmy, and Oam Patel for feedback.

In the comments of the post on Google Deepmind's CCS challenges paper, I expressed skepticism that some of the experimental results seemed possible. When addressing my concerns, Rohin Shah made some claims along the lines of “If an LLM linearly represents features a and b, then it will also linearly represent their XOR, <span>_aoplus b_</span>, and this is true even in settings where there's no obvious reason the model would need to make use of the feature <span>_aoplus b._</span>”

For reasons that I’ll explain below, I thought this claim was absolutely bonkers, both in general and in the specific setting that the GDM paper was working in. So I ran some experiments to prove Rohin wrong.

The result: Rohin was right and [...]

---

Outline:

(02:41) The RAX claim: LLMs linearly represent XORs of arbitrary features, even when there's no reason to do so

(05:01) RAX would be very surprising

(05:39) Without additional assumptions, RAX implies linear probes shouldn’t generalize

(08:44) Models have exponentially more stuff than you thought they did

(09:58) The evidence: RAX is true in every case I’ve checked

(13:30) What does RAX mean for model internals research?

(13:41) Qualitatively new reasons for linear probes not to generalize

(15:47) Results of probing experiments are much harder to interpret

(18:34) Applications of interpretability need to either have a way to distinguish XORs of features from basic features, or need to be robust to an exponential increase in number of features

(19:58) What's going on?

(20:21) Basic features are more salient than XORs

(21:28) Maybe _a\\oplus b_ is represented “indicentally” because NN representations are high-dimensional with lots of stuff represented by chance

(23:32) Maybe _a\\oplus b_ is represented “incidentally” because it's possible to aggregate noisy signals from many features which are correlated with boolean functions of a and b

(26:48) Maybe models track which features are basic and enforce that these features be more salient

(27:37) Models compute a bunch, but not all, XORs in a way that we don’t currently understand

---

First published:
January 3rd, 2024

Source:
https://www.lesswrong.com/posts/hjJXCn9GsskysDceS/what-s-up-with-llms-representing-xors-of-arbitrary-features

---

Narrated by TYPE III AUDIO.

“What’s up with LLMs representing XORs of arbitrary features?” by Sam Marks

LessWrong (30+ Karma)