Large clusters for small models — Daniel Svonava, Superlinked

Small open-source models provide frontier-level performance for specific tasks while drastically reducing costs and latency compared to proprietary managed services. Efficiently serving these models requires moving away from top-down routing, which creates bottlenecks, toward a centralized queue architecture where workers independently optimize batching. This approach maximizes GPU utilization and simplifies the integration of diverse model architectures and adapters like LoRAs. By utilizing an open-source infrastructure stack, organizations can achieve significant throughput improvements—such as processing hundreds of thousands of tokens per second for embeddings—on readily available, older hardware. This shift enables teams to maintain control over fine-tuned artifacts and scale workloads linearly with GPU count, effectively bypassing the limitations of restrictive, managed cloud AI platforms.