Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI

Routing AI inference in production requires balancing performance, reliability, and cache locality across geographically distributed GPU clusters. OpenAI’s inference load balancer (IRB) evolved from a feedback-loop-based system using weighted consistent hashing to a two-tier architecture. This system separates a control plane, which computes globally optimized routing weights based on engine health, capacity, and network latency, from a data plane that executes these decisions locally with minimal overhead. By incorporating KV cache awareness and real-time engine signals, the system minimizes end-to-end latency while avoiding oscillations common in traditional PID-based controllers. To maintain stability under stress, the architecture employs proactive protection mechanisms, including outlier penalties, dynamic retry budgets to prevent retry storms, and graceful load shedding when demand exceeds available capacity.