Audio note: this article contains 149 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

ARC recently released our first empirical paper: Estimating the Probabilities of Rare Language Model Outputs. In this work, we construct a simple setting for low probability estimation — single-token argmax sampling in transformers — and use it to compare the performance of various estimation methods. ARC views low probability estimation as a potential technique for mitigating worst-case behavior from AI, including deceptive alignment; see our previous theory post on estimating tail risks in neural networks for further discussion.

Outline of this post:

Problem statement and description of our setting
Estimation methods we compared: importance sampling and activation extrapolation
Summary of empirical results
Discussion, including motivation, limitations, related work, and future directions

[...]

---

Outline:

(01:32) Problem statement

(03:24) Estimation methods

(04:57) Importance sampling

(05:57) Independent Token Gradient Importance Sampling (ITGIS)

(07:16) Metropolis—Hastings Importance Sampling (MHIS)

(08:50) Activation extrapolation

(09:21) Quadratic Logit Decomposition (QLD)

(11:06) Gaussian Logit Difference

(11:41) Empirical results

(14:08) Discussion

(14:11) Motivation

(16:15) Importance sampling versus activation extrapolation

(17:15) Limitations

(18:54) Related work

(19:56) Future directions

The original text contained 7 footnotes which were omitted from this narration.

The original text contained 2 images which were described by AI.

---

First published:
October 18th, 2024

Source:
https://www.lesswrong.com/posts/Tzbvdp5rhM4L8aSKL/low-probability-estimation-in-language-models

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Low Probability Estimation in Language Models” by Jacob_Hilton

LessWrong (30+ Karma)

“Low Probability Estimation in Language Models” by Jacob_Hilton