Audio note: this article contains 225 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Introduction

There are <span>_208_</span> ways to output the text ▁LessWrong[1] with the Llama 3 tokenizer, but even if you were to work with Llama 3 for thousands of hours, you would be unlikely to see any but one. An example that generalizes quite widely: if you prompt Llama 3.2 3B Base with the text You're interested in rationality and AI? You should visit, there is a <span>_approx 22.7003%_</span> chance that it outputs the text ▁LessWrong, of which

<span>_approx 22.7001%_</span> is that it outputs exactly the tokens ▁Less and Wrong,
<span>_approx 0.00024%_</span> that it outputs exactly the tokens ▁Less, W, and rong,
and <span>_approx 0.000017%_</span> chance that it outputs any of the other <span>_206_</span> tokenizations which result in the text ▁LessWrong.