AI code benchmarks lied to us

Current AI coding benchmarks, particularly SWEbench Pro, fail to accurately measure model performance due to widespread data contamination, poor prompt design, and flawed verification processes. These metrics often produce misleading results, making inferior models appear competitive with state-of-the-art systems. DeepSWE addresses these systemic issues by utilizing novel, behavior-focused tasks that require end-to-end exploration rather than simple code execution. Data from this new benchmark reveals a significant performance gap between top-tier models like GPT-5.5 and open-weight alternatives, a disparity previously obscured by existing tests. Furthermore, analysis of token usage and operational costs demonstrates that many models are significantly less efficient and effective for real-world development than reported. Developers should prioritize creating custom, task-specific benchmarks based on their own recurring failures to better evaluate which agents actually solve complex, real-world coding problems.