Inferact: Building the Infrastructure That Runs Modern AI

The discussion centers on the increasing complexity of AI inference, contrasting it with the challenges of training models. Simon Mo and Woosuk Kwon, co-founders of Inferact and creators of vLLM, detail how the open-source inference engine addresses the growing demands of running large language models efficiently. They highlight the shift from static, standardized inputs in traditional machine learning to the dynamic, unpredictable nature of LLM requests. The conversation covers the importance of scheduling and memory management, particularly with the advent of AI agents requiring persistent states and external tool interactions. They emphasize the role of open-source in promoting diversity in models and hardware, enabling tailored solutions for specific use cases.