Yann Lecun: Meta AI, Open Source, Limits of LLMs, AGI & the Future of AI | Lex Fridman Podcast #416 | Lex Fridman

Yann LeCun, Meta's chief AI scientist, joins Lex Fridman to discuss the limitations of autoregressive large language models (LLMs) and proposes joint embedding predictive architectures (JEPA) as a more promising path toward advanced machine intelligence. LeCun argues that LLMs lack a true understanding of the physical world, persistent memory, and planning capabilities, unlike humans who learn through vast sensory input. He advocates for open-source AI development to ensure diversity and prevent control by a few entities, drawing parallels to the importance of a free and diverse press. LeCun expresses optimism about AI's potential to augment human intelligence, envisioning a future where AI assistants empower individuals, while acknowledging the need for guardrails to mitigate potential risks.

Outlines

Part 1: Open Source vs. Proprietary AI

Part 2: Limitations of Current LLMs

Part 3: JEPA and World Models

Part 4: Planning and Reasoning Architectures

Part 5: Addressing LLM Flaws

Part 6: Energy-Based Models and Optimization

Part 7: AI Ethics, Bias, and Open Source

Part 8: The Path to Human-Level AI

Part 9: Future Outlook and Advice

Sign in to continue reading, translating and more.

Open full episode in Podwise

Yann Lecun: Meta AI, Open Source, Limits of LLMs, AGI & the Future of AI | Lex Fridman Podcast #416

Lex Fridman

Part 1: Open Source vs. Proprietary AI

00:00The Danger of Concentrated Power in Proprietary AI Systems and Yann LeCun's Controversial Position

The Danger of Concentrated Power in Proprietary AI Systems and Yann LeCun's Controversial Position

Part 2: Limitations of Current LLMs

02:47LLMs Limitations: Understanding, Memory, Reasoning, and Planning

LLMs Limitations: Understanding, Memory, Reasoning, and Planning

05:56Grounding Intelligence in Reality: Language vs. Sensory Input and the Moravec Paradox

Grounding Intelligence in Reality: Language vs. Sensory Input and the Moravec Paradox

09:45The Inability of LLMs to Truly Understand the World

The Inability of LLMs to Truly Understand the World

13:46Abstract Thought vs. Token Prediction: Why LLMs Lack True Understanding

Abstract Thought vs. Token Prediction: Why LLMs Lack True Understanding

Part 3: JEPA and World Models

17:39Building World Models: The Limitations of Prediction and Generative Models

Building World Models: The Limitations of Prediction and Generative Models

23:07Joint Embedding Predictive Architecture (JEPA): A New Approach to Self-Supervised Learning

Joint Embedding Predictive Architecture (JEPA): A New Approach to Self-Supervised Learning

28:15JEPA vs. LLMs: Abstract Representation and the Importance of Redundancy

JEPA vs. LLMs: Abstract Representation and the Importance of Redundancy

32:30Combining Visual and Language Data: Avoiding the Temptation to Cheat

Combining Visual and Language Data: Avoiding the Temptation to Cheat

35:33Non-Contrastive Learning and Video Joint Embedding Predictive Architecture (VJEPA)

Non-Contrastive Learning and Video Joint Embedding Predictive Architecture (VJEPA)

Part 4: Planning and Reasoning Architectures

40:36Model Predictive Control: Planning Actions with a World Model

Model Predictive Control: Planning Actions with a World Model

44:19Hierarchical Planning: Decomposing Complex Actions into Sub-Goals

Hierarchical Planning: Decomposing Complex Actions into Sub-Goals

49:33The Power of Self-Supervised Learning and the Illusion of Fluency in LLMs

The Power of Self-Supervised Learning and the Illusion of Fluency in LLMs

55:00The Limits of LLMs and the Need for Joint Embedding

The Limits of LLMs and the Need for Joint Embedding

59:51Common Sense Reasoning and the Limitations of Language-Based Systems

Common Sense Reasoning and the Limitations of Language-Based Systems

Part 5: Addressing LLM Flaws

1:04:09Hallucinations in LLMs: The Exponential Accumulation of Errors

Hallucinations in LLMs: The Exponential Accumulation of Errors

1:07:31The Curse of Dimensionality and the Primitive Reasoning of LLMs

The Curse of Dimensionality and the Primitive Reasoning of LLMs

1:12:29System One vs. System Two: Planning and Reasoning Beyond Autoregressive Prediction

System One vs. System Two: Planning and Reasoning Beyond Autoregressive Prediction

Part 6: Energy-Based Models and Optimization

1:18:30Energy-Based Models: Optimization in Abstract Representation Space

Energy-Based Models: Optimization in Abstract Representation Space

1:25:16Training Energy-Based Models: Contrastive vs. Non-Contrastive Methods

Training Energy-Based Models: Contrastive vs. Non-Contrastive Methods

1:29:55Abandoning Generative Models and Embracing Model Predictive Control

Abandoning Generative Models and Embracing Model Predictive Control

Part 7: AI Ethics, Bias, and Open Source

1:32:31The Role of Human Feedback and the Inevitable Bias in AI Systems

The Role of Human Feedback and the Inevitable Bias in AI Systems

1:35:26Open Source as the Answer: Diversity and the Future of AI

Open Source as the Answer: Diversity and the Future of AI

1:43:28Meta's Business Model and the Benefits of Open Source

Meta's Business Model and the Benefits of Open Source

1:47:26The Ideological Lean of Big Tech and the Need for Diversity

The Ideological Lean of Big Tech and the Need for Diversity

1:53:17Guardrails and the Limits of AI's Harmful Potential

Guardrails and the Limits of AI's Harmful Potential

Part 8: The Path to Human-Level AI

1:57:25The Future of LLAMA and Yann LeCun's Excitement for AI

The Future of LLAMA and Yann LeCun's Excitement for AI

2:01:28The Importance of Hardware Innovation and Power Efficiency

The Importance of Hardware Innovation and Power Efficiency

2:04:21AGI is Not Coming Soon: Gradual Progress and the Importance of Guardrails

AGI is Not Coming Soon: Gradual Progress and the Importance of Guardrails

2:10:31The Fallacy of AI Dominance and the Importance of Submissiveness

The Fallacy of AI Dominance and the Importance of Submissiveness

2:12:27The Iterative Design of Guardrails and the Analogy to Turbojet Design

The Iterative Design of Guardrails and the Analogy to Turbojet Design

2:17:32The Cat and Mouse Game: Countermeasures and the Dissemination of Information

The Cat and Mouse Game: Countermeasures and the Dissemination of Information

Part 9: Future Outlook and Advice

2:20:24The Psychology of AI Doomers and the Fear of Change

The Psychology of AI Doomers and the Fear of Change

2:23:51The Power of AI and the Need for Open Source Platforms

The Power of AI and the Need for Open Source Platforms

2:27:44Trusting Humans and Institutions: Good AI vs. Bad AI