Complete Beginner's Course on AI Evaluations: Step by Step (2025) | Aman Khan

In this podcast episode, Aman Khan, Head of Product at Arise, joins the host to demonstrate how experienced product managers conduct AI evaluations using a real-world example. They discuss the importance of AI evaluations due to LLM hallucinations and outline four types of evaluations: code-based, human, LLM as a judge, and user evals. They walk through the process of defining an evaluation rubric, creating a golden dataset, and using LLM as a judge, using the example of building a customer support agent for an on-running shoe company. They emphasize the iterative nature of prompt engineering, manual evaluation, and the importance of aligning LLM evaluations with human judgment, and also touch on tools like Anthropic's Workbench and Arise for streamlining the evaluation process.