When I put on my LLM skeptic hat, sometimes I think things like “LLMs don’t really understand what they’re saying”. What do I even mean by that? What's my mental model for what is and isn’t going on inside LLMs minds?

First and foremost: the phenomenon precedes the model. That is, when interacting with LLMs, it sure feels like there's something systematically missing which one could reasonably call “understanding”. I’m going to articulate some mental models below, but even if I imagine all those mental models are wrong, there's still this feeling that LLMs are missing something and I’m not quite sure what it is.

That said, I do have some intuitions and mental models for what the missing thing looks like. So I’ll run the question by my intuitions a few times, and try to articulate those models.

First Pass: A Bag Of Map-Pieces

Imagine taking a map of the world, then taking a bunch of pictures of little pieces of the map - e.g. one picture might be around the state of Rhode Island, another might be a patch of Pacific Ocean, etc. Then we put all the pictures in a bag, and forget about the original [...]

---

Outline:

(01:02) First Pass: A Bag Of Map-Pieces

(02:05) Second Pass: Consistent Domains

(03:30) Third Pass: Aphantasia

(05:00) Fourth Pass: Noticing And Improving

The original text contained 1 footnote which was omitted from this narration.

---

First published:
November 8th, 2025

Source:
https://www.lesswrong.com/posts/trzFrnhRoeofmLz4e/insofar-as-i-think-llms-don-t-really-understand-things-what

---

Narrated by TYPE III AUDIO.

“Insofar As I Think LLMs ‘Don’t Really Understand Things’, What Do I Mean By That?” by johnswentworth

LessWrong (30+ Karma)