The End of GPU Scaling? Compute & The Agent Era — Tim Dettmers (Ai2) & Dan Fu (Together AI)

The debate over AGI centers on the tension between physical computational limits and the potential for massive scaling. Tim Dettmers contends that hardware is hitting diminishing returns, constrained by the von Neumann bottleneck and the physical limits of memory movement and precision. Conversely, Dan Fu posits that models are lagging indicators, noting that current hardware remains significantly underutilized and that substantial performance gains are possible through better kernel optimization and cluster scaling. Both experts agree that the practical utility of AI agents in coding and specialized tasks is already transforming productivity. They emphasize that mastering these tools requires deep domain expertise, as agents function most effectively when managed by skilled professionals. Looking toward 2026, the focus shifts from general AGI to the deployment of specialized, efficient small models and architectural innovations that bypass traditional transformer limitations.