Why Traditional Benchmarks Fail Modern AI Models with OpenAI Research Scientist Noam Brown

AI model evaluation currently suffers from a flawed reliance on static benchmarks that ignore the impact of test-time compute. As models like those developed at OpenAI demonstrate, performance is a function of the compute budget allocated to a task, meaning traditional benchmark grids fail to capture true capability. To address this, evaluations must shift toward plotting performance against variable compute budgets—such as tokens or time—rather than relying on single-number metrics. This is particularly critical for safety assessments, as models can exhibit dangerous capabilities only when given sufficient time to reason. While recursive self-improvement and AI-assisted research are accelerating, the necessity of large-scale test-time compute creates a practical bottleneck, suggesting a more gradual, compute-constrained evolution of AI capabilities rather than an immediate, overnight intelligence explosion.