Inference Deployments and Comms Implication by Cen Zhao, Xiaodong Wang, and Jianyu Huang

The podcast features Jianyu Huang, Xiaodong Wang, and Cen Zhao from Meta discussing LLM inference deployments and termination implications, particularly focusing on parallelism strategies to optimize performance. Jianyu introduces the basics of LLM inference, highlighting the pre-fill and decoding stages, and key performance metrics like cost, throughput, and latency. Cen then explains Tensor Parallelism and introduces the Direct Data Access (DDA) algorithm to improve all-reduce operations, showing performance gains in AMD launches. Jianyu returns to discuss Contextual Parallelism, including Interleaved Attention Layers (IROP) for long-context inference, and Xiaodong concludes with Expert Parallelism, detailing optimizations like dynamic all-to-all and persistent all-to-all to address communication bottlenecks, and they also discuss future challenges and opportunities in optimizing communication within kernels and cloud fabric design.