Databricks Co-Founder: Eval Limitations, Why China is Winning Open Source and Future of AI Infra

In this interview, Ion Stoica, co-founder of Databricks, AnyScale, and LLMarena, discusses the launch of LLMarena, a company focused on AI model evaluations, stemming from a project at Berkeley. Stoica explains the company's vision to create dynamic, user-preference-driven benchmarks to combat contamination in static benchmarks. The conversation covers the structural advantages China has in open-source model development due to closer collaboration between academia and industry, and the potential overbuilding of AI infrastructure in the US. Stoica also touches on the importance of reliability in AI applications, the challenges in generalizability of evaluation tools, and future trends in AI infrastructure, including vertically integrated co-design and optimization between networking and compute. Finally, Stoica shares his thoughts on the progress of open-source models, the slower-than-expected emergence of alternatives to NVIDIA, and the potential for AI to generate novel ideas in hard sciences, with the bottleneck being testing and validation.