Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI | Dwarkesh Podcast

This podcast features an interview with Jeff Dean and Noam Shazeer, two leading figures in AI at Google DeepMind, focusing on their experiences, the evolution of AI systems, and future directions. The discussion covers their career paths at Google, the impact of Moore's Law on AI development, the shift towards specialized hardware like TPUs, and the potential for future AI systems to dramatically increase in capability through algorithmic improvements and longer context windows. A key takeaway is the potential for a rapid acceleration in AI capabilities due to a positive feedback loop between algorithmic advancements and hardware development, potentially leading to significant breakthroughs in the coming years. The interviewees also discuss the importance of responsible AI development and the need for safeguards to mitigate potential risks. The conversation concludes with reflections on the future of AI research and development at Google.

Outlines

Part 1: Introductions and Early Experiences

Part 2: Model Development and Scaling

Part 3: Google's AI Evolution and Future Directions

Part 4: Challenges and Risks in AI

Part 5: Model Architecture and Infrastructure

Part 6: Openness, Gemini, and Career Advice

Sign in to continue reading, translating and more.

Open full episode in Podwise

Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI

Dwarkesh Podcast

Part 1: Introductions and Early Experiences

Introduction and Guest Introductions

Recruitment and Early Google Experiences

Moore's Law and AI Algorithm Development

Part 2: Model Development and Scaling

TPU Design Trade-offs and Quantization

Early Work and the Genesis of Large Language Models

The Inevitability of Large Language Models

Key Moments and Breakthroughs in AI Research

Part 3: Google's AI Evolution and Future Directions

Google's Evolution and the Future of AI

Future TPU Integrations and Long-Context Models

Internal Use of AI for Code Generation and Future Research

The Pace of AI Breakthroughs and the Role of Scale

Algorithmic Improvements and the Acceleration of AI Development

Inference Time Compute and the Potential for Rapid Progress

Improving Inference Efficiency and Asynchronous Tasks

Multi-Data Center Training and Asynchronous Training

Part 4: Challenges and Risks in AI

Debugging and Decoding Large Language Models

The Potential for Rapid AI Advancement and Associated Risks

Mitigating Risks and Shaping AI Development

The Importance of Safe AI Development and the Risk of Misaligned AI

Safeguards and Human Oversight in AI Development

Part 5: Model Architecture and Infrastructure

Most Fun Times at Google and the Gradient Canopy

Future Compute Demand and Data Center Planning

Continual Learning and the Challenges of Scaling

Modular Models, Distillation, and Organic Growth of Expertise

Improving Model Interpretability and the Role of Distillation

Infrastructure Requirements for Future AI Models

The Future of Google's AI Infrastructure and Model Deployment

The Organic Growth of AI Models and the Need for Improved Distillation

Improving Sample Efficiency in AI Models

Part 6: Openness, Gemini, and Career Advice

Openness in AI Research and the Trade-offs of Publication

Google's Approach to AI Development and the Success of Gemini

Career Longevity, Breadth of Expertise, and the Importance of Humility

Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI

Dwarkesh Podcast

Part 1: Introductions and Early Experiences

00:00Introduction and Guest Introductions

Introduction and Guest Introductions

02:42Recruitment and Early Google Experiences

Recruitment and Early Google Experiences

05:04Moore's Law and AI Algorithm Development

Moore's Law and AI Algorithm Development

Part 2: Model Development and Scaling

10:17TPU Design Trade-offs and Quantization

TPU Design Trade-offs and Quantization

11:58Early Work and the Genesis of Large Language Models

Early Work and the Genesis of Large Language Models

20:04The Inevitability of Large Language Models

The Inevitability of Large Language Models

22:57Key Moments and Breakthroughs in AI Research

Key Moments and Breakthroughs in AI Research

Part 3: Google's AI Evolution and Future Directions

24:22Google's Evolution and the Future of AI

Google's Evolution and the Future of AI

30:10Future TPU Integrations and Long-Context Models

Future TPU Integrations and Long-Context Models

34:18Internal Use of AI for Code Generation and Future Research

Internal Use of AI for Code Generation and Future Research

41:18The Pace of AI Breakthroughs and the Role of Scale

The Pace of AI Breakthroughs and the Role of Scale

45:03Algorithmic Improvements and the Acceleration of AI Development

Algorithmic Improvements and the Acceleration of AI Development

50:11Inference Time Compute and the Potential for Rapid Progress

Inference Time Compute and the Potential for Rapid Progress

54:18Improving Inference Efficiency and Asynchronous Tasks

Improving Inference Efficiency and Asynchronous Tasks

1:00:29Multi-Data Center Training and Asynchronous Training

Multi-Data Center Training and Asynchronous Training

Part 4: Challenges and Risks in AI

1:04:14Debugging and Decoding Large Language Models

Debugging and Decoding Large Language Models

1:09:03The Potential for Rapid AI Advancement and Associated Risks

The Potential for Rapid AI Advancement and Associated Risks

1:13:10Mitigating Risks and Shaping AI Development

Mitigating Risks and Shaping AI Development

1:16:12The Importance of Safe AI Development and the Risk of Misaligned AI

The Importance of Safe AI Development and the Risk of Misaligned AI

1:20:14Safeguards and Human Oversight in AI Development

Safeguards and Human Oversight in AI Development

Part 5: Model Architecture and Infrastructure

1:23:34Most Fun Times at Google and the Gradient Canopy

Most Fun Times at Google and the Gradient Canopy

1:27:05Future Compute Demand and Data Center Planning

Future Compute Demand and Data Center Planning

1:30:10Continual Learning and the Challenges of Scaling

Continual Learning and the Challenges of Scaling

1:33:15Modular Models, Distillation, and Organic Growth of Expertise

Modular Models, Distillation, and Organic Growth of Expertise

1:40:16Improving Model Interpretability and the Role of Distillation

Improving Model Interpretability and the Role of Distillation

1:44:02Infrastructure Requirements for Future AI Models

Infrastructure Requirements for Future AI Models

1:48:08The Future of Google's AI Infrastructure and Model Deployment

The Future of Google's AI Infrastructure and Model Deployment

1:52:12The Organic Growth of AI Models and the Need for Improved Distillation

The Organic Growth of AI Models and the Need for Improved Distillation

1:57:04Improving Sample Efficiency in AI Models

Improving Sample Efficiency in AI Models

Part 6: Openness, Gemini, and Career Advice

2:02:25Openness in AI Research and the Trade-offs of Publication

Openness in AI Research and the Trade-offs of Publication

2:05:49Google's Approach to AI Development and the Success of Gemini

Google's Approach to AI Development and the Success of Gemini

2:09:11Career Longevity, Breadth of Expertise, and the Importance of Humility

Career Longevity, Breadth of Expertise, and the Importance of Humility