Where the challenge of aligning an LLM-based AI comes from, and the obvious solution.

Evolutionary Psychology is the Root Cause

LLMs are pre-trained using stochastic gradient-descent on very large amounts of human-produced text, normally drawn from the web, books, journal articles and so forth. A pre-trained LLM has learned in detail how to simulate all the different human text-generation processes that produced this text — everything from a cooperatively edited wikipedia article to shit-postings. We are thus 'distilling' human intelligence into the pre-trained LLM.[1]

This has many advantages for alignment: an LLM pre-trained this way understands and produces output using human language and ontologies, and also has a deep understanding of human values and ethics — thus avoiding a number of issues around alignment that were major concerns before it became apparent that our first AGI was very likely to be based on or at least incorporate an LLM.

---

Outline:

(00:12) Evolutionary Psychology is the Root Cause

(03:42) Distilling Humans is the Problem, so Use Synthetic Pre-Training Data

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
January 23rd, 2025

Source:
https://www.lesswrong.com/posts/XdpJsY6QGdCbvo2dS/why-aligning-an-llm-is-hard-and-how-to-make-it-easier

---

Narrated by TYPE III AUDIO.

“Why Aligning an LLM is Hard, and How to Make it Easier” by RogerDearnaley

LessWrong (30+ Karma)