RAG Is Not Solved – Your Evaluation Just Sucks

In this YAAP Podcast episode, Yuval and Niv discuss the challenges and potential solutions in RAG (Retrieval-Augmented Generation) evaluation. Niv argues that current RAG benchmarks are flawed because they don't adequately represent real-world issues, particularly the way documents are chunked and indexed. He suggests that these benchmarks often optimize for local problems and fail to capture the diverse perspectives needed to answer complex questions. Niv proposes a "structured RAG" approach, which involves extracting information from documents and converting it into a structured format, such as an SQL database, to enable more reliable querying. The discussion also covers the trade-offs between memory, latency, and accuracy in RAG systems, emphasizing the need to understand the specific problem being solved before choosing a RAG implementation.