The goal of this post is to clarify a few concepts relating to AI Alignment under a common framework. The main concepts to be clarified:

Optimization. Specifically, this will be a type of Vingean agency. It will split into Selection vs Control variants.
Reference (the relationship which holds between map and territory; aka semantics, aka meaning). Specifically, this will be a teleosemantic theory.

The main new concepts employed will be endorsement and legitimacy.

TLDR:

Endorsement of a process is when you would take its conclusions for your own, if you knew them.
Legitimacy relates to endorsement in the same way that good relates to utility. (IE utility/endorsement are generic mathematical theories of agency; good/legitimate refer to the specific thing we care about.)
We perceive agency when something is better at doing something than us; we endorse some aspect of its reasoning or activity. (Endorse as [...]

---

Outline:

(01:28) Broader Context

(02:52) Meaning

(04:01) Teleosemantics

(05:32) Plurality of Teleosemantic Meaning

(06:57) Agency

(09:20) Endorsement

(14:11) Conditional Endorsement

(15:27) Radical Probabilism and Other Generalizations

(16:04) Legitimacy

(18:10) Updatelessness

(19:12) Questions and Conjectures

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
December 19th, 2023

Source:
https://www.lesswrong.com/posts/bnnhypM5MXBHAATLw/meaning-and-agency

---

“Meaning & Agency” by abramdemski

LessWrong (30+ Karma)