Stanford CS336 Language Modeling from Scratch | Spring 2025 | Lecture 12: Evaluation

The podcast discusses the complexities and challenges of evaluating language models, highlighting the "evaluation crisis" due to saturated or gamed benchmarks. It covers various evaluation methods, including benchmark scores (MMLU, AIME, etc.), cost analysis, user choice data, and human preferences. The speaker emphasizes that there is no one-size-fits-all evaluation, as the goal determines the approach. The discussion includes a framework for evaluation, considering inputs, prompting strategies, output assessment, and result interpretation, as well as touching on perplexity, instruction following, agent benchmarks, safety, and realism in evaluations.