Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)

In this episode of the podcast, Lenny interviews Hamel Husain and Shreya Shankar about evals, a systematic way to measure and improve AI applications. They discuss the importance of data analysis in identifying problems, categorizing them using AI, and then creating code-based or LLM-as-judge evals to test and monitor the AI's performance. They emphasize the need for human oversight in the eval process, the value of starting with error analysis, and the importance of aligning LLM-as-judge evals with human judgment. They also address common misconceptions about evals and offer tips for success, highlighting that evals are not just about catching bugs but about making AI products better and more profitable.