How to Scale AI Application Inference 100x ft. Fireworks’ Lin Qiao

Lin Qiao, CEO of Fireworks, outlines the future of AI inference as a multi-dimensional optimization problem focused on aligning model performance with specific application data distributions. Moving beyond off-the-shelf models, the next generation of inference requires co-optimizing quality, speed, and cost to drive down expenses by 10 to 100 times. This process involves managing a "combinatorial explosion" of over 100,000 variables, including hardware selection, token prediction strategies, and reinforcement tuning using production data. Fireworks addresses these complexities through a virtual cloud infrastructure that abstracts hardware management and provides developer tools for rapid scaling. Real-world applications of this approach include a food chain expanding AI features from one to 1,000 shops and a software company scaling from 100,000 to 25 million developers within three months, demonstrating the necessity of customized inference for sustainable business growth.