Are LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google

Reliable performance benchmarking for production-scale LLM inference requires addressing significant technical pitfalls often overlooked in standard testing tools. Common issues like Python’s Global Interpreter Lock (GIL) create artificial CPU bottlenecks, while inconsistent data set sampling and temperature settings lead to non-reproducible results. Benchmark harnesses frequently suffer from latency inflation, masking true system performance and complicating SLO compliance. To solve these challenges, the Inference Perf project introduces a multi-process architecture that ensures high-fidelity metrics and accurate load generation. By utilizing declarative configurations and a standardized workload catalog, this approach enables developers to simulate real-world production environments effectively. These methods allow for precise measurement of throughput and latency, ensuring that performance optimizations are validated against actual demand rather than being skewed by limitations within the benchmarking infrastructure itself.