AI Inference Is a Data Movement Problem: Microsoft's Saurabh Dighe on Maia 300

Microsoft’s Maia 300 architecture represents a strategic shift toward inference-first AI infrastructure, designed to maximize token throughput and energy efficiency for planet-scale workloads. Saurabh Dighe, Corporate Vice President of Azure Hardware Systems, details how the system leverages a software-defined data flow architecture to minimize energy-intensive data movement. By utilizing a multi-chiplet design on TSMC’s N2P process and integrating eight stacks of HBM4 memory, Maia 300 delivers 18 terabytes per second of bandwidth, enabling a 7x increase in token throughput per rack compared to its predecessor. The architecture addresses the evolving demands of agentic AI, which requires persistent KV caches and high memory capacity. Furthermore, the system incorporates flexible cooling solutions, allowing deployment in both liquid-cooled and air-cooled data centers, ensuring that Microsoft can scale its AI capabilities across diverse global infrastructure environments while maintaining strict power and cost constraints.