The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI

Agentic inference requires a fundamental shift in infrastructure design to support the transition of AI agents into massive production. Unlike traditional chat-based workloads, agentic tasks involve iterative loops of planning, tool execution, and observation, causing context to grow over time and necessitating the optimization of end-to-end task latency rather than individual request speed. FriendliAI addresses these challenges through a specialized inference cloud built on four pillars: prefix caching, efficient KV cache management, cache-aware routing, and agent-aware scheduling. By leveraging open-weight models, organizations can achieve frontier-level performance at a fraction of the cost associated with closed-source alternatives. This architecture significantly improves production reliability and speed, as demonstrated by coding tools like Cursor, which achieve faster execution and lower error rates by utilizing these optimized inference strategies for complex, long-horizon agentic workflows.