Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

[This is a mildly-edited version of a google doc I wrote at OpenAI in July 2022. I had intended to get it published in some form, but never got around to it for various reasons. I have now received approval to put it up as a blog post. The main thing of interest here is the distinctions I make; particularly the concept of self-location. Also the examples in the appendix that illustrate the distinctions. I lump all three concepts (self-knowledge, self-location, introspection) together under the banner of Self-Awareness, but since that's a spicy term which may have other connotations, these days I'd probably use the more neutral term Situational Awareness.]

Summary.

All three kinds of self-awareness come in degrees, but can be measured with appropriately designed evals/benchmarks. This doc explains how.
Self-knowledge: How much does the [...]

---

Outline:

(02:03) Outline:

(02:57) Self Knowledge

(03:00) What it means

(03:42) How to test for it

(04:23) Introspection

(04:27) What it means

(05:26) How to test for it

(09:53) Self-Location

(09:56) What it means

(12:08) How to test for it

(12:12) “Natural” Method: Look to see whether the model's performance/loss has a discontinuous improvement after it reads descriptions of its reward function and/or training environment.

(13:59) “Naive” Method: Just ask the model who it is.

(15:44) “Sophisticated” method: Experiment designed to directly test self-location ability.

(16:57) Importance

(17:01) Self-awareness → Consciousness → Moral Patienthood

(18:04) Self-awareness → Strategic Awareness and Agency → APS-AI

(19:29) Self-awareness → Situational Awareness → Alignment Failures

(20:51) Recommendations

(21:02) Appendix

(21:15) Self-knowledge + introspection but no self-location:

(22:50) Self-knowledge + self-location but no introspection:

(24:14) Introspection + self-location but little self-knowledge:

---

First published:
February 17th, 2024

Source:
https://www.lesswrong.com/posts/QEBFZtP64DdhjE3Sz/self-awareness-taxonomy-and-eval-suite-proposal

---

Narrated by TYPE III AUDIO.

“Self-Awareness: Taxonomy and eval suite proposal” by Daniel Kokotajlo

LessWrong (30+ Karma)