Feldera: Bridging Batch and Streaming with Incremental Computation

This episode explores Feldera, an incremental compute engine designed for continuous computation of data, ML, and AI workloads. Against the backdrop of traditional batch computing's inefficiencies (recomputing even with minimal data changes), Feldera introduces incremental computation, intelligently retaining past work to drastically speed up query processing. More significantly, the discussion highlights Feldera's unique capabilities, contrasting it with existing technologies like Materialize (sharing a common ancestor but pushing boundaries with DBSP, a novel mathematical foundation) and federated query engines (differentiating itself by offering a user-friendly SQL interface for both streaming and batch data). For instance, Feldera addresses the challenge of combining streaming and batch data sources, a common scenario in real-world analytics. The conversation further delves into Feldera's architecture, its use of Rust and Data Fusion, and its applications in machine learning, particularly real-time feature engineering. Finally, the episode concludes by discussing Feldera's open-source and enterprise offerings, its future roadmap (including scaling to larger datasets and integrating with object storage), and the broader implications of its approach for simplifying data management and streamlining change data capture.