Summary

When discussing the possibility that LLMs will cease to reason in transparent natural language with other AI safety researchers, we have sometimes noticed that we talk past each other: e.g., when discussing ‘neuralese reasoning’, some people have indefinitely long chains of recurrent activations in mind, while others think of an alien language in token-space. This post will propose terminology for distinguishing different types of hidden reasoning to create a shared language for such conversations.

In short, our taxonomy categorizes types of hidden reasoning as follows:

Neuralese: Arbitrary-depth reasoning in high-dimensional latent space, enabled by recurrence.
Hidden parallelized reasoning: Important steps are computed in activation-space in parallel without appearing in the chain-of-thought (CoT).
- Subcategories: partial verbalization, hidden utilization of test-time compute, epiphenomenal reasoning
Encoded reasoning: The AI either encodes the CoT itself in such a way that the monitor cannot decode, or encodes additional information inside seemingly legible [...]