The Hidden Challenges of Running AI at Scale in Production

CoreWeave's SVP of Engineering, Chen Goldberg, discusses the shift from AI experimentation to production across diverse industries. Companies outside Silicon Valley are leveraging their unique data assets to innovate, prompting a need for specialized AI cloud platforms. Goldberg highlights that companies engaged in AI training or inference for production workloads prioritize security, reliability, and scale, and should consider AI-first cloud platforms. She introduces CoreWeave's ARENA, a real infrastructure environment for benchmarking AI workloads, and emphasizes the importance of "goodput," the actual time GPUs spend on useful work. Goldberg also touches on the increasing accessibility of reinforcement learning and the growing adoption of AI agents, while advising companies to experiment with AI to avoid falling behind.