Why Inference is hard..

Large language models are not standalone executables but collections of artifacts requiring specialized inference engines to operationalize. Efficient model loading relies on memory mapping (mmap), which leverages the operating system to manage memory lazily, reducing initial latency and preventing system-wide memory exhaustion. Quantization serves as a critical compression layer, transforming high-precision weights into lower-resolution formats like INT4 or INT8. While basic methods like Round-to-Nearest can degrade accuracy, advanced techniques such as KQuants utilize hierarchical scaling, and AWQ identifies "salient weights" through activation magnitude to preserve model quality. Furthermore, specialized formats like EXL2 employ mixed precision and Hessian-based sensitivity analysis to optimize performance. These innovations enable large models to run on consumer-grade hardware by balancing the trade-offs between memory limitations, execution speed, and output accuracy.