Why Memory Is AI's Biggest Bottleneck with Vikram Sekar | EP 170

Memory serves as the fundamental bottleneck in AI development, dictating the evolution of both hardware infrastructure and software efficiency. As AI agents move toward broader adoption, the industry is shifting its focus from raw training throughput to inference-heavy workloads that demand more sophisticated memory hierarchies. Physical limitations of copper interconnects at speeds exceeding 400Gbps are driving a critical transition toward optical solutions and co-packaged optics to maintain data center scalability. Vikram Sekar, an expert in semiconductor and data center architecture, highlights that while researchers seek to optimize memory usage, the persistent Jevons paradox ensures that efficiency gains are quickly met with insatiable demand for more compute. Ultimately, the future of AI hinges on solving these memory-intensive challenges, as current architectures remain constrained by the physical realities of data movement and power delivery.