Ep 70: Karol Hausman and Danny Driess (Physical Intelligence) Unpack the Most Recent Breakthroughs & Path to Generalist Robots

In this episode of Unsupervised Learning, Jacob Effron interviews Karol Hausman, CEO and co-founder, and Danny Driess, a key researcher at Physical Intelligence, an AI robotics company. They discuss the evolution of AI robotics over the last 15 years, highlighting the shift from coding robots to learn tasks from experience, and the impact of transformer-based models. They share key moments, such as the RD2 experiment, that signaled breakthroughs in the field. The conversation covers the current state of robotics, focusing on capability, generalization, and performance, and addresses challenges like achieving human-level robustness. They also discuss the role of hardware, the comparison to self-driving cars, the importance of data collection, and future research directions, including generative video models and scaling laws. The episode concludes with quickfire questions about AI hype and predictions for the future of robotics in homes.

Outlines

Sign in to continue reading, translating and more.

Continue

Unsupervised Learning

Introduction to AI Robotics with Physical Intelligence

The Evolution of AI Robotics: From Code to Learning

Replicating Large Language Model Success in Robotics

Breakthrough Moments: RD2 and Connecting Internet Knowledge to Robotics

The Genesis of Physical Intelligence: A Focused Approach to Solving Physical Intelligence

Categorizing Progress: Capability, Generalization, and Performance

The Performance Threshold: Balancing Speed, Accuracy, and Human-Like Resilience

Robotics vs. Self-Driving Cars: Similarities and Fundamental Differences

Hardware vs. Intelligence: Identifying the Bottleneck in Robotics

Milestones in Intelligence: Generalization and the Magic of Unseen Environments

The Next Milestone: Achieving Robust Performance at Scale

Exceeding Expectations: The Rapid Progress in Robotics

The Impact of LLM Improvement on Robotics Timelines

Outstanding Research Questions: Generative Video Models and the Execution Problem

Scaling Laws for Robotics: Dollars to Model Capability

Infrastructure Needs: Data Storage, Pipelines, and Data Collection Strategies

Task Selection: Balancing Complexity, Understanding, and Data Collection

Evaluating Model Improvement: Guidelines, Statistical Analysis, and Simulation

The Role of Simulation: Evaluations and Diverse Real-World Data

Research Strategy: Balancing Exploration, Exploitation, and Taste

Identifying Taste: Past Performance, Soft Skills, and Willingness to Work

Open Source Strategy: Mitigating Risk and Inviting Collaboration

The Impact of Open Source: Diverse Applications and Real-World Learning

Training Models Faster: Knowledge Insulation and Tokenized Actions

Addressing Latency: Image In-Painting Techniques for Real-Time Adjustments

The Future of Task-Specific Training: Steerable and Promptable Base Models

Zero-Shot Performance: The Paradigm Shift in Pre-Trained Models

One Thing Changed: The Environment Scaling Curve

Overhyped and Underhyped: A Balanced Perspective

Underhyped: Generalist Robotics Models

Underhyped: Waymo

A Million Homes with Robots: A Five-Year Prediction

Vibe Coding for Hardware: A Cartoon-Like Future

The Future Without Chores: Enabling Human Potential and Contact Information

Ep 70: Karol Hausman and Danny Driess (Physical Intelligence) Unpack the Most Recent Breakthroughs & Path to Generalist Robots

Unsupervised Learning

00:00Introduction to AI Robotics with Physical Intelligence

Introduction to AI Robotics with Physical Intelligence

01:12The Evolution of AI Robotics: From Code to Learning

The Evolution of AI Robotics: From Code to Learning

03:46Replicating Large Language Model Success in Robotics

Replicating Large Language Model Success in Robotics

05:51Breakthrough Moments: RD2 and Connecting Internet Knowledge to Robotics

Breakthrough Moments: RD2 and Connecting Internet Knowledge to Robotics

07:36The Genesis of Physical Intelligence: A Focused Approach to Solving Physical Intelligence

The Genesis of Physical Intelligence: A Focused Approach to Solving Physical Intelligence

09:22Categorizing Progress: Capability, Generalization, and Performance

Categorizing Progress: Capability, Generalization, and Performance

11:18The Performance Threshold: Balancing Speed, Accuracy, and Human-Like Resilience

The Performance Threshold: Balancing Speed, Accuracy, and Human-Like Resilience

13:59Robotics vs. Self-Driving Cars: Similarities and Fundamental Differences

Robotics vs. Self-Driving Cars: Similarities and Fundamental Differences

16:15Hardware vs. Intelligence: Identifying the Bottleneck in Robotics

Hardware vs. Intelligence: Identifying the Bottleneck in Robotics

19:57Milestones in Intelligence: Generalization and the Magic of Unseen Environments

Milestones in Intelligence: Generalization and the Magic of Unseen Environments

22:20The Next Milestone: Achieving Robust Performance at Scale

The Next Milestone: Achieving Robust Performance at Scale

23:45Exceeding Expectations: The Rapid Progress in Robotics

Exceeding Expectations: The Rapid Progress in Robotics

25:51The Impact of LLM Improvement on Robotics Timelines

The Impact of LLM Improvement on Robotics Timelines

27:38Outstanding Research Questions: Generative Video Models and the Execution Problem

Outstanding Research Questions: Generative Video Models and the Execution Problem

29:54Scaling Laws for Robotics: Dollars to Model Capability

Scaling Laws for Robotics: Dollars to Model Capability

31:31Infrastructure Needs: Data Storage, Pipelines, and Data Collection Strategies

Infrastructure Needs: Data Storage, Pipelines, and Data Collection Strategies

34:40Task Selection: Balancing Complexity, Understanding, and Data Collection

Task Selection: Balancing Complexity, Understanding, and Data Collection

37:41Evaluating Model Improvement: Guidelines, Statistical Analysis, and Simulation

Evaluating Model Improvement: Guidelines, Statistical Analysis, and Simulation

40:18The Role of Simulation: Evaluations and Diverse Real-World Data

The Role of Simulation: Evaluations and Diverse Real-World Data

42:53Research Strategy: Balancing Exploration, Exploitation, and Taste

Research Strategy: Balancing Exploration, Exploitation, and Taste

44:26Identifying Taste: Past Performance, Soft Skills, and Willingness to Work

Identifying Taste: Past Performance, Soft Skills, and Willingness to Work

47:06Open Source Strategy: Mitigating Risk and Inviting Collaboration

Open Source Strategy: Mitigating Risk and Inviting Collaboration

49:11The Impact of Open Source: Diverse Applications and Real-World Learning

The Impact of Open Source: Diverse Applications and Real-World Learning

51:18Training Models Faster: Knowledge Insulation and Tokenized Actions

Training Models Faster: Knowledge Insulation and Tokenized Actions

54:06Addressing Latency: Image In-Painting Techniques for Real-Time Adjustments

Addressing Latency: Image In-Painting Techniques for Real-Time Adjustments

57:09The Future of Task-Specific Training: Steerable and Promptable Base Models

The Future of Task-Specific Training: Steerable and Promptable Base Models

59:04Zero-Shot Performance: The Paradigm Shift in Pre-Trained Models

Zero-Shot Performance: The Paradigm Shift in Pre-Trained Models

1:00:21One Thing Changed: The Environment Scaling Curve

One Thing Changed: The Environment Scaling Curve

1:01:14Overhyped and Underhyped: A Balanced Perspective

Overhyped and Underhyped: A Balanced Perspective

1:02:04Underhyped: Generalist Robotics Models

Underhyped: Generalist Robotics Models

1:03:00Underhyped: Waymo

Underhyped: Waymo

1:04:30A Million Homes with Robots: A Five-Year Prediction

A Million Homes with Robots: A Five-Year Prediction

1:05:28Vibe Coding for Hardware: A Cartoon-Like Future

Vibe Coding for Hardware: A Cartoon-Like Future

1:06:42The Future Without Chores: Enabling Human Potential and Contact Information

The Future Without Chores: Enabling Human Potential and Contact Information