Woosuk Kwon is CTO of Inferact and creator of the vLLM inference library. Woosuk shares what it takes to build the most popular open-source LLM inference engine from a human-centered perspective.

Outline:

0:00 - Prelude: Introducing Woosuk and Inferact

0:57 - Woosuk’s First PhD Project

3:57 - How the vLLM Project Got Started

7:15 - AI Infra Needs More Than Just Efficiency

12:05 - How AI Infra and Human-centered AI Are Connected

12:58 - How to Prioritize Feature Requests for Popular AI Infra

16:15 - Streaming Requests and Realtime API

23:02 - Multi-turn, Agentic, Proactive LLMs

25:00 - How to Design AI Infra in a Principled Way

27:10 - How to Design an AI Inference Engine for Continue Learning with RL

33:02 - Would LoRA Training Affect RL Infra Design?

36:05 - Why Start an AI Inference Infra Startup?

41:07 - What Effortless Inference with Open-source Models Means for Developers

44:38 - A Vision for On-device AI Inference

47:26 - Can Today’s Coding Agents Create vLLM?

References:

Inferact: https://inferact.ai/

Efficient Memory Management for Large Language Model Serving with PagedAttention: https://arxiv.org/abs/2309.06180

Streaming Requests & Realtime API in vLLM: https://vllm.ai/blog/streaming-realtime

RL’s Razor: Why Online Reinforcement Learning Forget Less: https://arxiv.org/abs/2509.04259

Podcast Links:

Podcast website: https://augmented-mind.github.io/

Apple Podcasts: https://podcasts.apple.com/us/podcast/augmented-mind-podcast/id1868102170

Spotify: https://open.spotify.com/show/40KculkYTe2tOpqJm6TAYr?si=PU_UncsMT4mXjVNCRwoXog&nd=1&dlsi=6d9bed7a43d64085

RSS: https://anchor.fm/s/10dbf5b7c/podcast/rss

About the Hosts:

The AM Podcast is hosted by Yijia Shao, Shannon Shen, and Michael Ryan, CS PhD students at Stanford University and MIT.

Sherry Wu is a professor at CMU whose research sits at the intersection of human-computer interaction and natural language processing. From making AI work for imperfect humans to making humans work better with AI — Sherry's work challenges us to rethink both sides of the equation.