Prior results have shown that LLMs released before 2024 can't leverage 'filler tokens'—unrelated tokens prior to the model's final answer—to perform additional computation and improve performance. [1] I did an investigation on more recent models (e.g. Opus 4.5) and found that many recent LLMs improve substantially on math problems when given filler tokens. That is, I force the LLM to answer some math question immediately without being able to reason using Chain-of-Thought (CoT), but do give the LLM filter tokens (e.g., text like "Filler: 1 2 3 ...") before it has to answer. Giving Opus 4.5 filler tokens [2] boosts no-CoT performance from 45% to 51% (p=4e-7) on a dataset of relatively easy (competition) math problems [3] . I find a similar effect from repeating the problem statement many times, e.g. Opus 4.5's no-CoT performance is boosted from 45% to 51%.

Repeating the problem statement generally works better and is more reliable than filler tokens, especially for relatively weaker models I test (e.g. Qwen3 235B A22B), though the performance boost is often very similar, especially for Anthropic models. The first model that is measurably uplifted by repeats/filler is Opus 3, so this effect has been around for a [...]

---

Outline:

(03:59) Datasets

(06:01) Prompt

(06:58) Results

(07:01) Performance vs number of repeats/filler

(08:29) Alternative types of filler tokens

(09:03) Minimal comparison on many models

(10:05) Relative performance improvement vs default absolute performance

(13:59) Comparing filler vs repeat

(14:44) Future work

(15:53) Appendix: How helpful were AIs for this project?

“Recent LLMs can use filler tokens or problem repeats to improve (no-CoT) math performance” by ryan_greenblatt

LessWrong (30+ Karma)