Prior work has examined 2-hop latent (by "latent" I mean: the model must answer immediately without any Chain-of-Thought) reasoning (e.g., "What element has atomic number (the age at which Tesla died)?") and found that LLM performance was limited aside from spurious successes (from memorization and shortcuts). I find that recent LLMs can now do 2-hop and 3-hop latent reasoning with moderate accuracy. I construct a new dataset for evaluating n-hop latent reasoning on natural facts (as in, facts that LLMs already know). On this dataset, I find that Gemini 3 Pro gets 60% of 2-hop questions right and 34% of 3-hop questions right. Opus 4 performs better than Opus 4.5 at this task; Opus 4 gets 31% of 2-hop questions right and 7% of 3-hop questions right. All models I evaluate have chance or near chance accuracy on 4-hop questions. Older models perform much worse; for instance, GPT-4 gets 9.7% of 2-hop questions right and 3.9% of 3-hop questions right.

I believe this new dataset I've created is the best existing dataset for evaluating n-hop latent reasoning. According to Balesni et al., prior datasets based on natural facts had issues with spurious successes while synthetic facts (introduced to [...]

---

Outline:

(06:22) Appendix: resolving Leo Gaos Manifold questions about fact composition

(10:40) Appendix: The effect of problem repeats and few-shots on these fact composition questions

(12:43) Appendix: adding the result of N 1-hop questions

(17:41) Appendix: Effects of using a different fact distribution

(18:31) Appendix: what 3-hop questions is Gemini 3 Pro getting right?

(21:51) Appendix: Dataset description

(30:02) Dataset sanity checks

(30:37) Appendix: AI usage

The original text contained 7 footnotes which were omitted from this narration.

---

“Recent LLMs can do 2-hop and 3-hop latent (no-CoT) reasoning on natural facts” by ryan_greenblatt

LessWrong (30+ Karma)