Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs

Chunking remains a critical yet under-optimized component of Retrieval-Augmented Generation (RAG) systems, despite claims that agentic search has rendered it obsolete. Fixed chunk sizes act as lossy compression, failing to account for the fact that optimal retrieval is inherently query-dependent. By implementing multiscale indexing—where data is indexed using multiple chunk sizes simultaneously—systems significantly improve recall. Aggregating these results via Reciprocal Rank Fusion (RRF) yields performance gains of 20% to 40% across diverse datasets, including QMSAM and Seinfeld. While this approach increases memory requirements by a constant factor, it maintains low latency through parallel retrieval execution. Rather than abandoning foundational infrastructure, developers can achieve substantial improvements in retrieval quality through simple, multi-layered indexing techniques that address the limitations of static, one-size-fits-all chunking strategies.