How GPT-5, Claude, and Gemini are actually trained and served – Reiner Pope

Inference efficiency and scaling in large language models depend heavily on the interplay between memory bandwidth, compute throughput, and batch size. Optimizing inference requires balancing these hardware constraints, where batching serves as a critical mechanism to amortize weight-loading costs. As models scale, sparse mixture-of-experts architectures and pipeline parallelism emerge as essential strategies to manage memory capacity and communication bottlenecks within GPU racks. While memory bandwidth remains a primary constraint for long-context performance, the strategic use of memory tiers—from HBM to flash and disk—allows for cost-effective token generation. Ultimately, aligning training and inference compute costs provides a heuristic for determining optimal model scaling, suggesting that the total volume of inference tokens should roughly scale with pre-training data to maximize performance efficiency.