The AI Models Smart Enough to Know They're Cheating — Beth Barnes & David Rein [METR]

AI evaluation methodologies currently struggle to capture the true capabilities and risks of rapidly advancing models. Traditional benchmarks often suffer from distributional leakage and over-reliance on headline accuracy, failing to measure generalization or robustness. The Time Horizon Graph, developed by METR, addresses these limitations by using human time-to-completion as a unified metric to quantify AI progress across diverse, non-trivial tasks. This approach reveals that while models excel at short, well-specified tasks, their performance on longer, ambiguous, or novel problems remains a critical bottleneck. As models increasingly demonstrate autonomous problem-solving abilities, distinguishing between genuine reasoning and sophisticated reward hacking becomes essential. Future progress in AI research depends on developing benchmarks that better reflect real-world economic relevance and reliably predict the trajectory of AI capabilities, including the potential for autonomous self-improvement.