How to Build Frontier-Lab Quality Evals with Daniel McKinnon, ex-PM at Meta, Google

Generative AI product management requires a fundamental shift from static Product Requirement Documents to "evals"—pre-baked sets of prompts and scoring criteria that define success through concrete examples. As AI models evolve from simple question-answering tools to complex, agentic systems, product managers must develop deep subject matter expertise to construct these evaluations. A successful eval follows a "Goldilocks" principle, providing enough difficulty to allow for iterative performance gains without rendering the task impossible. This process is critical for aligning engineering efforts with user needs, particularly in specialized fields like clinical genomics where AI must handle nuanced, multi-step reasoning. Beyond technical implementation, the industry is seeing a cultural pivot where high-conviction, founder-led organizations like Meta are aggressively adapting their product strategies to maintain dominance against competitors like Google in this rapidly changing AI landscape.