Audio note: this article contains 37 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Authors: Riya Tyagi, Daria Ivanova, Arthur Conmy, Neel Nanda

Riya and Daria are co-first authors. This work was largely done during a research sprint for Neel Nanda's MATS 9.0 training phase.

🖥️ Deployment code

⚙️ Interactive demo

TL;DR

We believe that more research effort should go into studying many chains of thought collectively and identifying global patterns across the collection.
We argue this direction is important and tractable; to start, we focus on a simpler setting: analyzing a collection of CoTs generated with a fixed model on a fixed prompt. We introduce two methods—semantic step clustering and algorithmic step clustering—that group recurring patterns across reasoning traces at different levels of abstraction.
Semantic step clustering leverages the fact that traces for the same prompt often repeat each other in slightly different ways. By clustering similar sentences like “1+2=3” and “2+1=3”, we compress a large set of CoTs into a detailed but messy reasoning graph.
- We evaluate semantic step clustering on a downstream task: predicting a model's output [...]

---

Outline:

(00:49) TL;DR

(03:03) Introduction

(06:59) Method 1: Semantic step clustering

(07:04) Building the semantic graph

(08:47) Chunking

(09:31) Clustering

(11:11) The issue of semantic drift

(12:30) Interlude: CoTs are chaotic!

(14:25) Sanity check: viewing our clusters

(15:29) Downstream task: predictive power

(16:54) Deriving predictions from our graph

(19:30) Predictive power baselines

(22:44) Results

(27:00) Method 2: Algorithmic step clustering

(27:58) Building the algorithm graph

(29:35) Sanity check: studying sample problems

(29:48) Tree structure

(30:45) Convergent structure

(31:37) Qualitative insights

(31:41) Models reuse strategies frequently

(33:24) Different models solve problems differently

(34:31) Takeaways and future work

(37:26) Acknowledgements

(37:41) Contribution Statement

(38:55) Appendix

(38:58) Chunking

(39:47) Clustering

(39:55) Stage 1: Agglomerative Clustering with Semantic Embedding Model

(41:01) Stage 2: LLM refinement via the Leiden Algorithm

(42:28) Predictive power

(42:31) Semantic graph prompts

(43:00) Method ablation study

(44:31) A comparison of three Gemini baselines:

(45:31) Gemini predictive power baseline prompts

(45:51) Gemini no rollout baseline (0 rollouts)

(46:13) Gemini 20 rollout baseline

(46:35) Gemini full context baseline (1000 rollouts)

(46:58) Footnotes

The original text contained 15 footnotes which were omitted from this narration.

---

First published:
January 13th, 2026

Source:
https://www.lesswrong.com/posts/q9g9zuudd3Pvw2cbj/global-cot-analysis-initial-attempts-to-uncover-patterns-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Figure 1: Overview of our two methods for mapping a model’s reasoning on a fixed prompt: semantic step clustering and algorithmic step clustering. Both methods chunk all rollouts into discrete pieces and cluster similar pieces to make a graph; individual rollouts become walks on this graph. Note: organic human-made figure.

Figure 2: Illustration of the chunking phase of the semantic graph pipeline.

Figure 3: Illustration of the clustering phase of the semantic graph pipeline.

Figure 4: Sketch of the resulting semantic graph.

Figure 5: Example of a group of chunks that got clustered together in the original version of our method. In this chain, every two neighboring chunks are similar to each other, but the first chunk has very little in common with the last one.

Figure 6: Embedding of a semantic clustering graph made with 100 rollouts on the “tricky sisters” prompt via gpt-oss-20b. Each rollout is depicted in a different color.

Figure 7: Same graph as above, but with only one correct rollout shown.

Figure 8: Embedding of a semantic clustering graph made with 150 rollouts on the “eval awareness” prompt via Claude Sonnet 4.5. Rollouts that exhibited evaluation awareness (as judged by 4o-mini) are colored red; rollouts that did not are blue.

Figure 9: Three example clusters for the

Figure 10: Diagram of how we derive predictions. A: slide a window over the rollout’s clusters and add 1 minus the cluster’s entropy for each cluster in the longest common subsequence. The rollout score is the maximum window score. B: The predicted correctness percentage is the weighted average of the 20 top-scoring rollouts.

Figure 11: In this plot, each colored point is a test prompt, and the black triangle shows the mean RMSE across prompts for each method. Methods are ordered left-to-right by the mean RMSE (best to worst).

$Figure 12: In this plot, each color represents a different baseline, with the bar numbers showing how many prefixes (aggregated across prompts) the RMSE is averaged over. We have three buckets for determinism, which we calculate as <span>__T3A_INLINE_LATEX_PLACEHOLDER___\min(p_{\text{correct}},p_{\text{incorrect}})___T3A_INLINE_LATEX_END_PLACEHOLDER__</span> for a prefix: 0-10%, 10-30%, and 30-50%.$

Figure 13: Illustration of how we get algorithmic step keywords from Gemini for the algorithm graph.

Figure 14: Illustration of how we split rollouts into algorithmic steps via keyword matching.

Figure 15: Sketch of the resulting algorithm graph.

Figure 16: A depiction of the tree-like structure we uncovered on “debugging.”

Figure 17: A depiction of the convergent structure we found on “minimum-square.”

Figure 18: Here, we have an example cue dictionary for the hex problem. For simplicity, we only include a sample of the keywords in the cue dictionary here.

Figure 19: An example labeled rollout for the hex question.We observe 4 algorithmic step switches in the same rollout!

Figure 20: A depiction of algorithm graphs for two models on “cubes.” The graph from Claude Sonnet 3.7 is on the left, and the graph from gpt-oss-20b is on the right.

Figure 21: Illustration of the “granularity” and “supervision” axes. Our methods are on opposite extremes with nothing in between.

Figure 22: Flowchart detailing our final chunking pipeline. We perform some generalsteps for all text, and then additional steps for text that contains computations.

Figure 23: Results of the one-by-one ablation study on the procedure for obtaining predictions from our graph, showing the RMSE for every ablation on each of the six test prompts. The gray line marks the average RMSE of the current method.

Figure 24: Results of a comparison of 0-rollout, 20-rollout,and 1000-rollout Gemini baselines on “bagel.”

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Global CoT Analysis: Initial attempts to uncover patterns across many chains of thought” by Riya Tyagi, daria, Arthur Conmy, Neel Nanda

LessWrong (30+ Karma)