Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer

Stateful LLM inference systems often fail silently, producing high-confidence gibberish rather than explicit errors. Debugging these issues requires moving beyond standard crash logs to forensic analysis of log-probabilities and kernel behavior. Two critical failures in the Jamba model’s Mamba architecture illustrate this: a scheduling error where decode operations preceded prefill, causing the model to process stale state data, and an integer overflow in CUDA kernels where 32-bit index pointers wrapped around during high-memory usage. By implementing log-probability comparison scripts against baseline implementations and stress-testing with constrained GPU memory, developers can isolate these elusive bugs. Ultimately, maintaining visibility into the inference lifecycle—such as propagating request IDs through the forward pass—is essential for diagnosing complex, non-crashing failures in production environments.