This post contains some rough reflections on the alignment community trying to make its ontology legible to the mainstream ML community, and the lessons we should take from that experience.

Historically, it was difficult for the alignment community to engage with the ML community because the alignment community was using a fundamentally different ontology—featuring concepts like inner vs outer alignment, mesa-optimizers, corrigibility, situational awareness, and so on. Even a concept as simple as "giving an AI an instruction in natural language" threw a kind of type error in the ML ontology, which thought of an AI's goals as determined by its loss function not its training data.

The concept of situational awareness is another one which doesn't really make sense in the classic ML ontology. My impression is that Ajeya giving a talk at OpenAI about situational awareness was one of the main drivers of Ilya getting into alignment. Unfortunately, Ilya's subsequent research on weak-to-strong generalization stayed pretty stuck in the ML ontology, which in my opinion made it unpromising from the get-go. (I don't remember if I stated this publicly at the time, but I was pretty critical internally inside OpenAI, especially to Collin Burns. In hindsight I [...]

---

First published:
February 24th, 2026

Source:
https://www.lesswrong.com/posts/Yz4YHncz2vwN4ksDA/the-ml-ontology-and-the-alignment-ontology

---

Narrated by TYPE III AUDIO.

“The ML ontology and the alignment ontology” by Richard_Ngo

LessWrong (30+ Karma)