It always feels wrong when people post chats where they ask an LLM questions about its internal experiences, how it works, or why it did something, but I had trouble articulating why beyond a vague, "How could they possibly know that?"[1]. This is my attempt at a better answer:

AI training data comes from humans, not AIs, so every piece of training data for "What would an AI say to X?" is from a human pretending to be an AI. The training data does not contain AIs describing their inner experiences or thought processes. Even synthetic training data only contains AIs predicting what a human pretending to be an AI would say. AIs are trained to predict the training data, not to learn unrelated abilities, so we should expect an AI asked to predict the thoughts of an AI to describe the thoughts of a human pretending to be [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
August 8th, 2025

Source:
https://www.lesswrong.com/posts/Af649z8maCD5mvDy6/what-would-a-human-pretending-to-be-an-ai-say

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Excuse the bad photoshop and inconsistent style, but I couldn't get Gemini/Imagen to one-shot

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“What would a human pretending to be an AI say?” by Brendan Long

LessWrong (30+ Karma)

“What would a human pretending to be an AI say?” by Brendan Long