What Comes After ChatGPT? The Mother of ImageNet Predicts The Future

In this episode of the Latent Space Podcast, Fei-Fei Li and Justin Johnson of WorldLabs discuss spatial intelligence and their new model, Marble, which generates explorable 3D worlds from texture images. They explore the differences between spatial and language intelligence, the importance of structure in 3D world modeling, and the role of open science versus proprietary models in AI development. They also touch on the challenges of resourcing academic AI research and the potential of using physics engines to enhance world models, as well as the shift in academia's role in AI, advocating for wacky ideas and theoretical underpinnings. The conversation further explores the capabilities and potential use cases of Marble, including its applications in gaming, VFX, film, robotics training, and interior design, while also considering the future of world models and the integration of physics and dynamics.

Outlines

Part 1: Genesis of WorldLabs and Marble

Part 2: Vision-Language Modeling and Real-Time Captioning

Part 3: World Models and Spatial Intelligence

Part 4: Conclusion and Future

Sign in to continue reading, translating and more.

Continue

The a16z Show

Part 1: Genesis of WorldLabs and Marble

Introduction to WorldLabs and the Genesis of Marble

The Evolution of Compute and the Open Science Model

Resourcing Academia and the Role of Open Datasets

Part 2: Vision-Language Modeling and Real-Time Captioning

Wacky Ideas and Hardware Scaling Limits

Early Work in Image Captioning and Vision-Language Modeling

Language Modeling and Dense Captioning

Real-Time Captioning Demo and the Necessity of Language

Part 3: World Models and Spatial Intelligence

Inductive Bias and Understanding in World Models

Human vs. AI Understanding and the Vision of Spatial Intelligence

Marble: A Generative Model of 3D Worlds

Atomic Units and the Data Structure of a World Model

Adding Physics and Dynamics to Gaussian Splats

Embodied Use Cases and Spatial vs. Traditional Intelligence

Spatial Intelligence and Embodied Experience

The Underappreciation of Vision and Spatial Intelligence

Multimodal Models and the Limits of LLMs

Learning Paradigms and Theory Building

Questioning Sequence-to-Sequence Modeling and the Nature of Transformers

Part 4: Conclusion and Future

Conclusion and Call to Action

What Comes After ChatGPT? The Mother of ImageNet Predicts The Future

The a16z Show

Part 1: Genesis of WorldLabs and Marble

00:00Introduction to WorldLabs and the Genesis of Marble

Introduction to WorldLabs and the Genesis of Marble

04:42The Evolution of Compute and the Open Science Model

The Evolution of Compute and the Open Science Model

07:19Resourcing Academia and the Role of Open Datasets

Resourcing Academia and the Role of Open Datasets

Part 2: Vision-Language Modeling and Real-Time Captioning

11:24Wacky Ideas and Hardware Scaling Limits

Wacky Ideas and Hardware Scaling Limits

14:14Early Work in Image Captioning and Vision-Language Modeling

Early Work in Image Captioning and Vision-Language Modeling

17:00Language Modeling and Dense Captioning

Language Modeling and Dense Captioning

20:03Real-Time Captioning Demo and the Necessity of Language

Real-Time Captioning Demo and the Necessity of Language

Part 3: World Models and Spatial Intelligence

23:40Inductive Bias and Understanding in World Models

Inductive Bias and Understanding in World Models

27:07Human vs. AI Understanding and the Vision of Spatial Intelligence

Human vs. AI Understanding and the Vision of Spatial Intelligence

31:38Marble: A Generative Model of 3D Worlds

Marble: A Generative Model of 3D Worlds

34:52Atomic Units and the Data Structure of a World Model

Atomic Units and the Data Structure of a World Model

38:10Adding Physics and Dynamics to Gaussian Splats

Adding Physics and Dynamics to Gaussian Splats

41:20Embodied Use Cases and Spatial vs. Traditional Intelligence

Embodied Use Cases and Spatial vs. Traditional Intelligence

44:36Spatial Intelligence and Embodied Experience

Spatial Intelligence and Embodied Experience

47:50The Underappreciation of Vision and Spatial Intelligence

The Underappreciation of Vision and Spatial Intelligence

50:50Multimodal Models and the Limits of LLMs

Multimodal Models and the Limits of LLMs

54:01Learning Paradigms and Theory Building

Learning Paradigms and Theory Building

57:05Questioning Sequence-to-Sequence Modeling and the Nature of Transformers

Questioning Sequence-to-Sequence Modeling and the Nature of Transformers

Part 4: Conclusion and Future

1:01:03Conclusion and Call to Action

Conclusion and Call to Action