This is a crosspost of my ICLR 2026 blogpost track post. All code and experiments are available at github.com/andyrdt/iclr_induction.

Summary

Park et al., 2025 show that when large language models (LLMs) process random walks on a graph, their internal representations come to mirror the underlying graph's structure. The authors interpret this broadly, suggesting that LLMs can "manipulate their representations in order to reflect concept semantics specified entirely in-context". In this post, we take a closer look at the underlying mechanism, and suggest a simpler explanation. We argue that induction circuits (Elhage et al., 2021; Olsson et al., 2022), a well-known mechanism for in-context bigram recall, suffice to explain both the task performance and the representation geometry observed by Park et al.

Recapitulation and reproduction of Park et al., 2025

We begin by describing the experimental setup of Park et al., 2025 and reproducing their main results on Llama-3.1-8B.

Figure 1. Overview of Park et al.
(a) The grid tracing task uses a 4×4 grid of words. (b) Models observe random walks on the grid (e.g., apple bird milk sand sun plane opera ...) where consecutive words are always neighbors. As the sequence length grows, the model begins to predict valid [...]