This is a linkpost for https://youtu.be/qV_rOlHjvvs

This video recounts an incident that occurred at OpenAI in which flipping a single minus sign led the RLHF process to make GPT-2 only output sexually explicit continuations.

The incident is described in OpenAI's paper "Fine-Tuning Language Models from Human Preferences" under section 4.4: "Bugs can optimize for bad behavior".

The script has been written by Jai, with some significant input and rework by me. You can read it below.

In 2019, one OpenAI researcher made a typo - and birthed an evil AI hell-bent on making everything as horny as possible.

This is the absurd, ridiculous, and yet true story of how it happened.

Part I: GPT

Since 2017, OpenAI has been building Generative Pre-trained Transformer models, or GPTs - language AIs with a singular focus on predicting text, trained across billions of writing samples. If you prompt a GPT model with [...]

---

Outline:

(00:48) Part I: GPT

(03:15) Part II: Human Feedback

(06:48) Part III: Rise of EvilHornyGPT

(09:50) Part IV: Aftermath

---

First published:
January 18th, 2024

Source:
https://www.lesswrong.com/posts/xD3wymX24BpqezBpw/the-true-story-of-how-gpt-2-became-maximally-lewd

Linkpost URL:
https://youtu.be/qV_rOlHjvvs

---

Narrated by TYPE III AUDIO.

[Linkpost] “The True Story of How GPT-2 Became Maximally Lewd” by Writer

LessWrong (30+ Karma)