Dreamers and Doomers: Jeremy Nixon at AGI House – #105 | Manifold

Jeremy Nixon, an AI entrepreneur, discusses the evolution and future of AI with Steve Hsu. Nixon argues that foundation models have already achieved AGI, disagreeing with the apocalyptic framing of AI risk prevalent in some rationalist communities. He traces the history of AI safety concerns, noting Elon Musk's early fears about DeepMind and the origins of OpenAI. Nixon critiques the "heroic ideology" driving existential risk concerns, suggesting a focus on "P-Life"—maximizing the probability of everyone living—rather than solely minimizing existential threats. He envisions a future where AI drives scientific progress, particularly in personalized medicine, but acknowledges potential societal and political obstacles to technological advancement.

Outlines

Part 1: Defining AGI and the Future of Physics

Part 2: Jeremy Nixon’s Journey and Silicon Valley Culture

Part 3: Critiquing the Rationalist and AI Safety Movements

Part 4: Infinity and the Automation of Science

Part 5: Action vs. Stagnation in AI Development

Part 6: Risk Assessment and "P-Life"

Part 7: Societal Impact and Industry Dynamics

Part 8: Leadership and the Race to AGI

Part 9: Technical Omnicompetence and Personal Philosophy

Sign in to continue reading, translating and more.

Open full episode in Podwise

Dreamers and Doomers: Jeremy Nixon at AGI House – #105

Manifold

Part 1: Defining AGI and the Future of Physics

00:00The original goals of AGI and the advent of foundation models

The original goals of AGI and the advent of foundation models

02:03Dreamers, Doomers, and the potential of AI in physics discovery

Dreamers, Doomers, and the potential of AI in physics discovery

Part 2: Jeremy Nixon’s Journey and Silicon Valley Culture

05:15Jeremy Nixon's path from Harvard to Google Brain and AI entrepreneurship

Jeremy Nixon's path from Harvard to Google Brain and AI entrepreneurship

08:34The AGI House communal living experiment and Silicon Valley culture

The AGI House communal living experiment and Silicon Valley culture

12:14The origins of OpenAI and Anthropic from Bay Area apocalyptic co-beliefs

The origins of OpenAI and Anthropic from Bay Area apocalyptic co-beliefs

Part 3: Critiquing the Rationalist and AI Safety Movements

19:23Defining AGI: Generality, foundation models, and recursively self-improving AI

Defining AGI: Generality, foundation models, and recursively self-improving AI

22:28The Rationalist Movement, cognitive biases, and the heroic ideology of saving the world

The Rationalist Movement, cognitive biases, and the heroic ideology of saving the world

28:40Utilitarian calculations, existential risk, and the creator complex

Utilitarian calculations, existential risk, and the creator complex

33:17The religious roots of the AI existential risk community and the pursuit of the machine god

The religious roots of the AI existential risk community and the pursuit of the machine god

Part 4: Infinity and the Automation of Science

38:01The flow of resources into AI and the opportunities for AI-native companies

The flow of resources into AI and the opportunities for AI-native companies

40:44Jeremy Nixon's vision for Infinity: Automating research and scientific progress

Jeremy Nixon's vision for Infinity: Automating research and scientific progress

43:14Discovering new foundation model training processes and the AI scientist paradigm

Discovering new foundation model training processes and the AI scientist paradigm

45:50The fragility of AI and the need for recalibration in the machine learning paradigm

The fragility of AI and the need for recalibration in the machine learning paradigm

Part 5: Action vs. Stagnation in AI Development

50:17The evolution of AI safety perspectives and the purity contest within the rationality community

The evolution of AI safety perspectives and the purity contest within the rationality community

53:07The importance of real-world action and the limitations of blog posts in the Doomer community

The importance of real-world action and the limitations of blog posts in the Doomer community

55:30The unique combination of intellectualism and practical action at AGI House

The unique combination of intellectualism and practical action at AGI House

58:07China's progress and the stagnation hypothesis in the West

China's progress and the stagnation hypothesis in the West

1:01:25The importance of practical problems and the need for moral credit

The importance of practical problems and the need for moral credit

Part 6: Risk Assessment and "P-Life"

1:03:11Uncertainty estimation and the limits of superintelligence

Uncertainty estimation and the limits of superintelligence

1:05:34The real-world risks of heart disease, cancer, and the miscalibration of diverse perspectives

The real-world risks of heart disease, cancer, and the miscalibration of diverse perspectives

1:08:49Conceptual criticisms of alignment and the redefinition of AGI

Conceptual criticisms of alignment and the redefinition of AGI

1:13:37Replacing P-Doom with P-Life: Maximizing the probability that everyone lives

Replacing P-Doom with P-Life: Maximizing the probability that everyone lives

1:17:06The low P-Life and the neolithic-style movements inhibiting bioprogress

The low P-Life and the neolithic-style movements inhibiting bioprogress

Part 7: Societal Impact and Industry Dynamics

1:21:53The bioethicists and the application of basic logic

The bioethicists and the application of basic logic

1:23:48The societal effects of job loss and the structural nature of most jobs

The societal effects of job loss and the structural nature of most jobs

1:27:20The pessimistic view of progress and the government inefficiency

The pessimistic view of progress and the government inefficiency

1:31:30The incentive structures and the regulation of AI

The incentive structures and the regulation of AI

1:35:55The lack of competition and the default state of affairs

The lack of competition and the default state of affairs

Part 8: Leadership and the Race to AGI

1:37:45The adventure-like terms and the construction of meaning processes

The adventure-like terms and the construction of meaning processes

1:40:46The heads of the top AI companies and the complex social environment

The heads of the top AI companies and the complex social environment

1:46:02The race to AGI and the conceptual confusion

The race to AGI and the conceptual confusion

1:47:39The dynamics by which monopoly is kind of inconceivable

The dynamics by which monopoly is kind of inconceivable

1:50:35The ideological perspective and the probabilities of risk

The ideological perspective and the probabilities of risk

Part 9: Technical Omnicompetence and Personal Philosophy

1:51:21The AGI House and the grounding belief

The AGI House and the grounding belief

1:53:56The vibe coding and the hyperstition event