AI Evaluations Crash Course in 50 Minutes (2025) | Hamel Husain

In this interview, Hamel Husain discusses AI evaluations with Peter Yang, using NurtureBoss, an AI-powered property management assistant, as a practical example. Hamel emphasizes the importance of manually reviewing traces (records of user interactions with the AI) and categorizing issues before applying automated evaluations. He advises against relying solely on generic metrics like "helpfulness" or agreement scores, advocating instead for identifying specific problems and measuring true positive and true negative rates to assess the accuracy of LLM judges. Hamel also shares insights into synthetic data generation and the value of building internal tools for data annotation, highlighting that a deep understanding of the data is crucial for effective AI evaluation and product improvement.