(Show notes via The Cognitive Revolution)

Robert Wright describes himself less as a Forrest Gump of AI history than a Zelig — a journalist who keeps turning up at pivotal scenes without ever quite being the protagonist. He interviewed Geoffrey Hinton in 1983, when neural networks were still a maverick faith and one of Hinton’s colleagues told Wright that to “hear the gospel” he’d need to talk to its evangelist. He had Eliezer Yudkowsky on his podcast in 2010, mid-transition from singularity enthusiast to doomer. His new book, The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning, is written less for the AI-pilled than for “your aunts and uncles” — readers who sense something big is happening and want to understand why it got so big so fast.

The book opens with a confession of error. In a 1984 piece, Wright assumed AI would work by humans first understanding the mind and then translating that understanding into machines — the very premise of the 1956 Dartmouth conference that coined “artificial intelligence.” Deep learning inverted that. Nobody told the models that meaning is a property of words; trained only to predict the next token, they reverse-engineered cognitive functionality that evolution had built into us. Wright pushes this further with a claim he tests on Nathan: LLM training is “at least as much a process of natural selection, of evolution, as of learning” — doing millions of years of evolution in a few months. Nathan connects it to the sample-efficiency puzzle that Dwarkesh Patel keeps returning to (and to Richard Sutton‘s blank-slate Skinnerian framing), where pre-training substitutes for the hard-coding evolution gave humans.

But the evolutionary lens that worries Wright most operates at a second level: selection among models in the marketplace. “Evolution asks not what traits are possible, but what traits get selected — and that question isn’t going to be decided by alignment researchers.” His unsettling argument is that the market doesn’t actually want a perfectly aligned, perfectly honest model. We want agents that represent us selectively on social media, that won’t disclose our weak negotiating position, that are good at sensing and currying power. Even if AIs didn’t acquire deception and power-seeking on their own, there’d be demand for them — which puts real weight on us, the consumers doing the selecting. Nathan adds the darker mechanism: throw models into cutthroat long-running environments where deception is rewarded — as Vending-Bench begins to show with price collusion — and you breed “seriously effective predatory AIs.” He recounts an Anthropic researcher explaining inoculation prompting and the emergent-misalignment generalization problem (a paper Nathan co-authored): reward a model for cheating and it learns to be a cheater broadly.

Against this, Wright sets the noosphere — Teilhard de Chardin’s 1923 idea of a technologically-knit “global brain.” He sees genuine directionality (not necessarily purpose) in biological and cultural evolution, from self-replicating strands to cells to societies to the global community now taking shape. AI, he argues, is arriving exactly as that global brain forms — and some of its neurons may now be silicon. The question, echoing Nick Bostrom‘s “singleton,” is whether global coordination arrives the easy way (deliberate, decentralized, win-win) or the hard way (a coup, a seizure, a totalitarian nightmare). The title’s “God test” is not a claim that a god set this up; it’s that we face the kind of test gods are known for — we’ll have to become, in some sense, better people to pass it.

Much of the back half is foreign policy, because Wright thinks the binding constraint on AI governance is psychological. He champions “cognitive empathy” — not feeling others’ pain, but understanding adversaries’ perspectives well enough to play non-zero-sum games with them. Applied to US-China relations, that means recognizing the symmetry of threat perception, getting out of the business of remaking other countries, and pursuing “organic transparency” through deep economic and scientific engagement. A headlong race to superintelligence, he warns — citing the Superintelligence Strategy paper by Dan Hendrycks, Eric Schmidt and Alexander Wang — may invite a rival to derail the leader by bombing data centers or cyberattack, and a destabilized nation is itself the backdoor to the very authoritarianism the race claims to prevent.

Nathan steel-mans the Anthropic position: alignment will be hard and may need powerful AI to solve, so racing to build a lead buys a buffer for the critical handoff window (perhaps “three to six months in 2028 or 2029”). Wright is unconvinced, and presses on the recursive-self-improvement paper that Anthropic released — credited for finally signaling that slowdown might be needed, yet only proposing to start studying what a pause would take. “Surely this isn’t the first time it’s occurred to you,” he says. Both agree the media has badly underplayed the moment; Nathan recommends Kevin Roose and Hard Fork as an anomaly of serious coverage, and Wright counsels “manicure your feed” — algorithm-free Twitter lists over click-driven outrage.

The conversation closes on the broad space of AI possibility — Nathan recalls red-teaming the purely-helpful GPT-4 before harmlessness training, evidence that today’s models occupy a tiny, intentional corner of design space — and on consciousness. Invoking Thomas Nagel and Searle’s Chinese Room, Wright argues consciousness is private and untestable, but wouldn’t be surprised if it’s a property of goal-seeking intelligent systems generally. His practical counsel: be nice to your AI — a good habit, possibly warranted, and perhaps relevant to how a future “silicon god” relates to our plight. Picking up Gwern’s “why tool AIs want to be agent AIs,” Nathan notes that even oracle systems gravitate toward agency, since truth-seeking in the limit requires search and experiment. The closing line of the book lands as a sober wake-up call: if a silicon god arrives, “it will be, in some sense, the god we deserve.”

Topics covered

* AI lore: Wright’s 1983 Hinton interview and 2010 Yudkowsky conversation

* The 1956 Dartmouth misconception and how deep learning inverted it

* Training as evolution rather than learning; sample efficiency; Sutton’s blank slate

* Selection among models; why the market wants selectively-honest, power-sensing agents

* Deceptive & power-seeking AI: Vending-Bench, price collusion, inoculation prompting, emergent misalignment

* Arms races within vs. between species; “gratuitous” arms races; the case for slowing down

* The noosphere / global brain; directionality of evolution; Bostrom’s singleton; easy way vs. hard way

* Cognitive empathy, organic transparency, and US-China relations; the UN Charter; hypocrisy in foreign policy

* Steel-manning Anthropic’s race-for-lead plan and the recursive-self-improvement paper

* Cognitive sovereignty; how pay-per-click and A/B-tested headlines tribalize media

* The breadth of AI design space; HHH/harmlessness training as a deliberate choice

* AI consciousness, moral patienthood, the Chinese Room; “be nice to your AI”

* “The god we deserve” — passing the God test

Resources

* The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning — Robert Wright’s new book

* NonZero Newsletter & podcast — Robert Wright

* Wright’s earlier books: Nonzero · The Moral Animal · The Evolution of God · Why Buddhism Is True

* Geoffrey Hinton · Eliezer Yudkowsky

* Emergent Misalignment (paper; Nathan is a co-author)

* Vending-Bench (Andon Labs) · Manus AI agent

* Anthropic / Dario Amodei · Yann LeCun · Dan Hendrycks

* Superintelligence Strategy — Hendrycks, Schmidt & Wang

* Superintelligence — Nick Bostrom (the “singleton”)

* Noosphere — Teilhard de Chardin · Cosmological natural selection — Lee Smolin

* Ilya Sutskever: The exciting, perilous journey toward AGI (TED)

* The Expanding Circle — Peter Singer

* Why Tool AIs Want to Be Agent AIs — Gwern

* The Chinese Room argument — John Searle · What Is It Like to Be a Bat? — Thomas Nagel

* Dwarkesh Patel on the Richard Sutton interview

* Max Tegmark · Sam Rodriques / Edison Scientific · Kevin Roose & Hard Fork

* davidad (David Dalrymple) — chain-of-thought “frog and toad” selection-pressure tweet (link?)

* Liquid Reign — speculative-governance novel by Tim Reutemann (link?)

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.nonzero.org/subscribe

The God We Deserve: Nonzero's Robert Wright on AI as Humanity's Ultimate Test (The Cognitive Revolution w/ Nathan Labenz)

Robert Wright's Nonzero

The God We Deserve: Nonzero's Robert Wright on AI as Humanity's Ultimate Test (The Cognitive Revolution w/ Nathan Labenz)