Reiner Pope – The math behind how LLMs are trained and served

AI inference economics hinge on the interplay between batch size, memory bandwidth, and compute throughput. Increasing batch size effectively amortizes weight-loading costs, but optimal performance requires balancing compute capacity with memory bandwidth, which often acts as the primary bottleneck for frontier models. While pipeline parallelism and expert parallelism enable scaling across multiple GPU racks, they introduce communication overheads that necessitate careful architectural alignment. The cost structure of modern API providers reflects these physical constraints, where memory bandwidth limitations drive pricing tiers for long-context inputs. Ultimately, achieving cost-efficiency across the model lifecycle—from pre-training and reinforcement learning to inference—requires equalizing compute expenditures, as the massive scale of inference traffic now rivals the total data volume used in pre-training.