(Originally published on No Set Gauge.)

Gustave Moreau, The Frogs Asking For A King

In the course of building a better world, people ask each other many questions. Which things should be managed by the government and which left to the market? What sort of technology, if any, is so dangerous that it should be kept secret, access curtailed, or development avoided? Is goodness fundamentally about following the right rules, achieving the right outcomes, or having the right character?

Reasonable people have different opinions on all these questions. But recently, Silicon Valley has seen lively debate on a question you’d hope was all too obvious: should humanity continue existing?

The idea that it shouldn’t was named successionism by Andrew Critch, and is motivated by the speed and power of AI development.

Some examples:

Already back in 2013, Elon Musk, freaked out by Demis Hassabis's warnings about AI risk, got into an argument with Larry Page about whether it matters if AI replaces humanity. Page called it just the next stage of evolution and those that resist it “speciesists”. Elon, who has often had good instincts on goals but is not known for his eloquence, retorted “Well [...]