I've been thinking about memetically fit Schelling points in the AGI policy space. I'll describe four such "Schelling policies", and use them as pedagogical examples.

Shut it all down

MIRI's new stated objective is the clearest example of a Schelling policy: "Shut it all down". MIRI states that they want governments to coordinate to pause all AI research that involves smarter-than-human systems. Laypeople will find this policy easy to understand, since they can rely on the shared cultural knowledge of CFC bans and international nuclear disarmament as case studies.

If you want to coordinate a large number of people coherently towards furthering a particular policy, "you get about five words" that you can make 'common knowledge' such that people can coordinate in a specific direction. The ease of communicating the policy makes a big difference in such conditions.

When you attempt to communicate an idea [...]

---

Outline:

(00:17) Shut it all down

(02:28) Effective Accelerationism (e/acc)

(04:58) We must beat China

(06:58) Only One Org

(09:36) Parting words

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
June 26th, 2024

Source:
https://www.lesswrong.com/posts/JECQZAXWbtGJdBuAC/schelling-points-in-the-agi-policy-space

---

Narrated by TYPE III AUDIO.

“Schelling points in the AGI policy space” by mesaoptimizer

LessWrong (30+ Karma)