How to look at your data — Jeff Huber (Choma) + Jason Liu (567)

Jeff Huber and Jason Liu present a two-part session on how to effectively analyze data to improve AI systems. Jeff discusses the importance of using fast evals—query and document pairs—to quickly and inexpensively measure retrieval system performance and suggests using LLMs to generate realistic queries. He shares a case study with Weights and Biases, demonstrating how different embedding models perform using both ground truth and synthetically generated queries. Jason then explains how to analyze the outputs of AI systems, such as chatbot conversations, to identify user needs and improve product development. He introduces Cura, a library for summarizing and clustering conversations to extract valuable metadata, enabling data-driven decisions on tool development and prompt engineering. The goal is to understand user behavior, segment user types, and make impact-weighted decisions to enhance AI application performance and guide product roadmaps.