Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)

In this episode of Lenny's Podcast, Lenny interviews Hamel Husain and Shreya Shankar about evals, a systematic way to measure and improve AI applications. They discuss the importance of data analysis, error analysis, and the role of LLMs in the eval process. They also address common misconceptions about evals, such as the belief that AI can fully automate the process or that buying a tool is sufficient, and emphasize the need for human judgment and domain expertise. They share practical tips and best practices for building effective evals, including the concept of a "benevolent dictator," the use of open coding and axial coding, and the importance of aligning LLM-as-judge with human expectations.