Computer Use at the Edge of the Statistical Precipice — Pierluca D'Oro, Programma Labs

Computer use agent (CUA) evaluation currently suffers from deterministic benchmarks that are easily gamed by "replay agents"—scripts that blindly replicate successful trajectories. These static environments render metrics like PASSAT-K misleading, as they fail to account for true model capability. To address this, the PRISM principles prioritize building robust, diverse, and verified environments, exemplified by the DigiWorld benchmark, which supports millions of unique, validated task configurations. Beyond environment design, rigorous evaluation must incorporate proper uncertainty quantification. Relying on simple rollouts often produces overconfident confidence intervals, leading to significant financial risks when deploying models. Establishing reliable statistical methods to capture both action-based and environmental stochasticity is essential for making informed, cost-effective decisions in model selection and deployment.