NVIDIA's AI Engineers: Agent Inference at Planetary Scale and "Speed of Light" — Nader Khalil (Brev), Kyle Kranen (Dynamo) | Latent Space: The AI Engineer Podcast

The Latent Space podcast features Nader Khalil and Kyle Kranen from NVIDIA, discussing developer experience, GPU technology, and the company's internal culture. They recount Brev's acquisition by NVIDIA, emphasizing the shared goal of simplifying developer access to GPUs, and touch on NVIDIA's developer experience initiatives. The conversation explores Dynamo, a data center scale inference engine, detailing its role in optimizing inference at scale through techniques like disaggregation, prefill, and decode. They also discuss "SOL" (Speed of Light) as a concept for creating urgency and understanding theoretical limits, and explore the balance between stability and innovation. The podcast further examines the evolution of AI, the importance of hardware-model co-design, and the potential of agents in coding and business applications.

Outlines

Part 1: Security, Acquisition, and Culture

Part 2: Developer Strategy and GPU Access

Part 3: Innovation and Market Creation

Part 4: Scaling Inference and Dynamo

Part 5: Hardware-Model Co-Design

Part 6: Agents in Production

Part 7: Coding Agents and CLIs

Part 8: Future Trends and Community

Sign in to continue reading, translating and more.

Open full episode in Podwise

NVIDIA's AI Engineers: Agent Inference at Planetary Scale and "Speed of Light" — Nader Khalil (Brev), Kyle Kranen (Dynamo)

Latent Space: The AI Engineer Podcast

Part 1: Security, Acquisition, and Culture

00:00Agent Security: Balancing Access and Enforcement to Prevent Malware Injection

Agent Security: Balancing Access and Enforcement to Prevent Malware Injection

00:38NVIDIA's Acquisition of Brev: A Developer-Focused Tool for GPU Access

NVIDIA's Acquisition of Brev: A Developer-Focused Tool for GPU Access

04:41Brev's Marketing Stunts and NVIDIA's Passion for Developer Experience

Brev's Marketing Stunts and NVIDIA's Passion for Developer Experience

07:19Acquisition Story: Brev's Integration into NVIDIA and Developer Experience Investment

Acquisition Story: Brev's Integration into NVIDIA and Developer Experience Investment

Part 2: Developer Strategy and GPU Access

09:15NVIDIA's Developer Experience Strategy: Expanding Access and Simplifying Complex Technology

NVIDIA's Developer Experience Strategy: Expanding Access and Simplifying Complex Technology

12:41NVIDIA Sync and SOL: Simplifying GPU Access and Defining Urgency

NVIDIA Sync and SOL: Simplifying GPU Access and Defining Urgency

15:16SOL Implementation: Prioritization, Stability, and Incremental Progress

SOL Implementation: Prioritization, Stability, and Incremental Progress

Part 3: Innovation and Market Creation

18:38Kyle's Journey at NVIDIA: From Recommenders to Graph Neural Networks

Kyle's Journey at NVIDIA: From Recommenders to Graph Neural Networks

21:14Passion-Driven Innovation and Market Creation at NVIDIA

Passion-Driven Innovation and Market Creation at NVIDIA

24:28Zero-Billion-Dollar Markets and Deep Learning Algorithms at NVIDIA

Zero-Billion-Dollar Markets and Deep Learning Algorithms at NVIDIA

Part 4: Scaling Inference and Dynamo

26:35Dynamo: NVIDIA's Data Center Scale Inference Engine

Dynamo: NVIDIA's Data Center Scale Inference Engine

28:47Scaling Inference: Overcoming Hardware and Algorithmic Bounds

Scaling Inference: Overcoming Hardware and Algorithmic Bounds

31:12Model Selection and the Three Axes of Inference: Quality, Cost, and Latency

Model Selection and the Three Axes of Inference: Quality, Cost, and Latency

34:44The "Try Again" Approach and Inference Reading Groups

The "Try Again" Approach and Inference Reading Groups

36:07Nemotron and Dynamo: Layered Releases and Inference Optimizations

Nemotron and Dynamo: Layered Releases and Inference Optimizations

Part 5: Hardware-Model Co-Design

38:42Disaggregation: Separating Prefill and Decode for Inference Optimization

Disaggregation: Separating Prefill and Decode for Inference Optimization

43:01Context Length Upper Bounds and Model Hardware Co-Design

Context Length Upper Bounds and Model Hardware Co-Design

45:50Sparsity, Attention Heads, and Hardware Model Co-Design

Sparsity, Attention Heads, and Hardware Model Co-Design

47:15Training Models with Harnesses and the Quest for Generality

Training Models with Harnesses and the Quest for Generality

51:21Unhobblers and the Future of Prefill and Decode

Unhobblers and the Future of Prefill and Decode

Part 6: Agents in Production

53:08GTC Sessions and the Future of Agents in Production Inference

GTC Sessions and the Future of Agents in Production Inference

55:25NVIDIA's Deployment of Codecs and the Fluidity of the Organization

NVIDIA's Deployment of Codecs and the Fluidity of the Organization

58:41Agent Security and NVIDIA's Internal Resources

Agent Security and NVIDIA's Internal Resources

1:01:14NVIDIA's Internal Tools and the Idea of a Fork VS Code Hackathon

NVIDIA's Internal Tools and the Idea of a Fork VS Code Hackathon

Part 7: Coding Agents and CLIs

1:03:34World's Shortest Hackathon and the Power of Coding Agents

World's Shortest Hackathon and the Power of Coding Agents

1:07:13The Importance of CLIs and Security Considerations

The Importance of CLIs and Security Considerations

1:09:38The Year of Coding Agents Breaking Containment and the Rent-a-Human Concept

The Year of Coding Agents Breaking Containment and the Rent-a-Human Concept

Part 8: Future Trends and Community

1:12:30The Year of the Sub-Agent and Speculative Decoding

The Year of the Sub-Agent and Speculative Decoding

1:14:05System as Model and the Insatiable Demand for Tokens

System as Model and the Insatiable Demand for Tokens

1:16:16Agent Run Time and Domain-Specific Desires

Agent Run Time and Domain-Specific Desires

1:17:40Speculative Decoding and the Human Equivalent Work Metric

Speculative Decoding and the Human Equivalent Work Metric

1:19:36San Francisco's Supportive Community and the Importance of Authenticity

San Francisco's Supportive Community and the Importance of Authenticity