Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

This is an attempt to distill a model of AGI alignment that I have gained primarily from thinkers such as Eliezer Yudkowsky (and to a lesser extent Paul Christiano), but explained in my own terms rather than attempting to hew close to these thinkers. I think I would be pretty good at passing an ideological Turing test for Eliezer Yudowsky on AGI alignment difficulty (but not AGI timelines), though what I'm doing in this post is not that, it's more like finding a branch in the possibility space as I see it that is close enough to Yudowsky's model that it's possible to talk in the same language.

Even if the problem turns out to not be very difficult, it's helpful to have a model of why one might think it is difficult, so as to [...]

---

Outline:

(01:15) Defining human values

(04:08) Normative criteria for AI

(06:20) Consequentialism is instrumentally useful for problem-solving

(08:14) Simple but capable AI methods for solving hard abstract problems are likely to model the real world

(11:02) Natural, consequentialist problem-solving methods that understand the real world may care about it

(12:26) Sometimes, real-world performance is what is desired

(14:15) Alignment might not be required for real-world performance compatible with human values, but this is still hard and impacts performance

(15:42) Myopic agents are tool-like

(17:39) Short-term compliance is instrumentally useful for a variety of value systems

(18:52) General agents tend to subvert constraints

(21:33) It is hard to specify optimization of a different agents utility function

(23:48) There are some paths forward

(27:38) Conclusion

---

First published:
December 31st, 2023

Source:
https://www.lesswrong.com/posts/wnkGXcAq4DCgY8HqA/a-case-for-ai-alignment-being-difficult

---

Narrated by TYPE III AUDIO.

“A case for AI alignment being difficult” by jessicata

LessWrong (30+ Karma)

“A case for AI alignment being difficult” by jessicata