Continual Learning Is the Next Bottleneck | Rohan Anil (Core Automation )

Rohan Anil, a former Google Brain researcher and co-founder of Core Automation, explores critical bottlenecks in current AI research, specifically focusing on the limitations of existing optimization methods and the necessity for more efficient, automated kernel generation. The discussion highlights the evolution from traditional linear solvers to modern neural network training, emphasizing how second-order methods like Shampoo can significantly accelerate convergence. Anil argues that while current coding agents excel at search, they often lack the fundamental understanding of hardware and numerical stability required for production-grade systems. He advocates for a shift toward models that co-evolve with research tasks, enabling continual learning and more effective use of computational resources. By combining human expertise with agentic systems, researchers can move beyond current scaling plateaus to achieve more reliable, efficient, and creative AI development.