[Note: this began life as a "Quick Takes" comment, but it got pretty long, so I figured I might as well convert it to a regular post.]

In LM training, every token provides new information about "the world beyond the LM" that can be used/"learned" in-context to better predict future tokens in the same window.

But when text is produced by autoregressive sampling from the same LM, it is not informative in the same way, at least not to the same extent[1]. Thus, sampling inevitably produces a distribution shift.

I think this is one of the reasons why it's (apparently) difficult to get instruction-tuned / HH-tuned models to report their uncertainty and level of competence accurately, rather than being overconfident.

(I doubt this is a novel point, I just haven't seen it spelled out explicitly before, and felt like doing so.)

Imagine that you read the following (as the [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
September 5th, 2024

Source:
https://www.lesswrong.com/posts/JeuAk53QfWauaGWfS/instruction-tuning-and-autoregressive-distribution-shift

---

Narrated by TYPE III AUDIO.

“instruction tuning and autoregressive distribution shift ” by nostalgebraist

LessWrong (30+ Karma)