Why Traditional Benchmarks Fail Modern AI Models with OpenAI Research Scientist Noam Brown

Modern AI model evaluations fail to accurately reflect capabilities because they ignore the critical role of test-time compute. As models like 5.5 demonstrate, performance on complex reasoning tasks—such as solving mathematical conjectures—scales significantly with increased inference budgets, rendering static, one-size-fits-all benchmark grids obsolete. The industry currently operates in a suboptimal equilibrium, prioritizing standardized reporting over metrics that account for the cost or time invested in a response. Noam Brown, a pioneer in AI reasoning, emphasizes that as models become more capable, the primary bottleneck for both research breakthroughs and safety assessments is the lack of standardized, compute-aware evaluation frameworks. Future progress requires shifting toward plotting performance as a function of inference budget, ensuring that evaluations capture the true potential of frontier models rather than just their performance at arbitrary, low-cost settings.