Inference Engineering 101 - with Philip Kiely

Inference engineering bridges the gap between raw AI model intelligence and production-ready performance. Philip Kiely, an Inference Engineer at BaseTen, explains that achieving efficient model deployment requires navigating complex trade-offs between compute-bound prefill phases and memory-bound decoding processes. Key optimization strategies include quantization—specifically transitioning to formats like NVFP4—and leveraging mixture-of-experts architectures to reduce hardware requirements. While large-scale data centers utilize disaggregated inference to maximize throughput, local self-hosting remains a distinct challenge involving hardware constraints and specialized parallelism. The conversation highlights how continuous advancements in model architecture and hardware, such as NVIDIA’s Blackwell stack, enable developers to push performance limits. Ultimately, the field is shifting from academic benchmarks toward binary production-readiness, where specialized optimizations significantly reduce the cost and energy required to serve high-demand AI applications.