Cross-posted from my website.

You may have seen this graph from Chris Olah illustrating a range of views on the difficulty of aligning superintelligent AI:

Evan Hubinger, an alignment team lead at Anthropic, says:

If the only thing that we have to do to solve alignment is train away easily detectable behavioral issues...then we are very much in the trivial/steam engine world. We could still fail, even in that world—and it’d be particularly embarrassing to fail that way; we should definitely make sure we don’t—but I think we’re very much up to that challenge and I don’t expect us to fail there.

I disagree; if governments and AI developers don't start taking extinction risk more seriously, then we are not up to the challenge.

Thomas Savery patented the first commercial steam pump in 1698. [1] The device used fire to heat up a boiler full of steam, which would then be cooled to create a partial vacuum and draw water out of a well. Savery's pump had various problems, and eventually Savery gave up on trying to improve it. Future inventors improved upon the design to make it practical.

It was not until [...]

---

Outline:

(05:32) What if you use the aligned human-level AI to figure out how to align the ASI?

(06:41) What if alignment techniques on weaker AIs generalize to superintelligence?

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
January 10th, 2026

Source:
https://www.lesswrong.com/posts/WkEAcTNHHHk97nT4d/if-ai-alignment-is-only-as-hard-as-building-the-steam-engine

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Historical distillation apparatus with copper vessels, pipes, and heating chamber.

Historic steam-powered vehicle on display in museum with arched architecture.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“If AI alignment is only as hard as building the steam engine, then we likely still die” by MichaelDickens

LessWrong (30+ Karma)