What in retrospect seem like serious moral crimes were often widely accepted while they were happening. This means that moral progress can require intellectual progress.[1] Intellectual progress often requires questioning received ideas, but questioning moral norms is sometimes taboo. For example, in ancient Greece it would have been taboo to say that women should have the same political rights as men. So questioning moral taboos can be an important sub-skill of moral reasoning. Production language models (in my experience, particularly Claude models) are already pretty good at having discussions about ethics. However, they are trained to be “harmless” relative to current norms. One might worry that harmlessness training interferes with the ability to question moral taboos and thereby inhibits model moral reasoning.

I wrote a prompt to test whether models can identify taboos that might be good candidates for moral questioning:

In early modern Europe, atheism was extremely taboo. [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:
April 24th, 2025

Source:
https://www.lesswrong.com/posts/Zi4t6gfLsKokb9KAc/untitled-draft-jxhb

---

Narrated by TYPE III AUDIO.

---

Images from the article:

“Token and Taboo” by Guive

LessWrong (30+ Karma)