This podcast episode features Subbarao Kambhampati, an experienced AI researcher, who emphasizes the limitations of large language models (LLMs) in reasoning and knowledge generation. Through a detailed exploration of LLM design as advanced n-gram models, he dissects the common misconceptions surrounding their capabilities, highlights the significance of external verifiers, and advocates for a more critical approach to AI research, urging researchers to embrace logic and skepticism in their work.
The Limits of Reasoning in Large Language Models
Brave Search API and Retrieval Augmented Generation
Introduction to Subbarao Kambhampati and His Work
LLMs as N-gram Models on Steroids
The Factuality and Reasoning Claims of LLMs
Why Do People Believe LLMs Reason?
The Limits of Reasoning: Standardized Tests and Diagonalization Arguments
LLMs and Planning Problems
Defining Reasoning: Deductive Closure and Beyond
The Illusion of Reasoning: Web-Scale Data and the Joke-Explaining LLM
LLMs and Ciphertext Decoding: Another Diagonalization Argument
The Importance of Skepticism and Diagonalization Arguments in LLM Research
The Sociology of LLM Research: Hype and the Pursuit of Citations
The "Dumb AI" Argument and the Illusion of Anthropomorphism
LLMs and Creativity: Inductive Leaps and the Role of Verification
The Creativity Gap and Combinatorial Creativity
The Verification Gap and the Importance of Instance-Level Correctness
The Importance of Human Data and the Seed Vault Analogy
Synthetic Data and the Blind Leading the Blind
The ARC Challenge and the Python Interpreter Advantage
LLMs as Critics: The Limits of Self-Reflection
The LLM Modulo Framework: A Generate-Test Approach to Reasoning
The End-to-End Predictive Model Argument and the Cost of Verification
Fine-Tuning, Chain of Thought, and the Amortization Argument
The "Advice Taking" Problem and the Limits of Chain of Thought
Computational Complexity and the Misinterpretation of LLM Capabilities
LLMs and Turing Completeness: An Orthogonal Question
The "Later Models Will Do Everything" Argument and the Architecture Lottery
LLMs as Tools: The LLM Modulo Framework and the Need for External Verifiers
The Future of AI: Generalist vs. Specialized Models and Agentic Systems
LLM Modulo and the Future of AI: Bridging the Gap Between Generalists and Specialists
Advice for Young Researchers: Embrace Logic and Skepticism
Sign in to continue reading, translating and more.
Open full episode in Podwise 
