47 - David Rein on METR Time Horizons

AI agent capability evaluation centers on measuring progress through the time required for humans to complete specific tasks, providing a unified metric to track AI performance over time. Research from METR reveals an exponential increase in the complexity and duration of tasks that AI models can successfully complete, with current models capable of handling tasks requiring roughly two hours of human effort. This capability growth, particularly in software engineering, data analysis, and cybersecurity, suggests a potential shift toward AI-driven research and development, raising concerns about rapid, recursive self-improvement. While current models demonstrate significant progress, evaluating their performance remains challenging due to the subjectivity of task difficulty and the high cost of human-calibrated benchmarks. Future research aims to refine these metrics, improve the ecological validity of evaluations, and better understand the interaction between AI systems and human productivity.