#562: DuckLake: The Lakehouse That's Just SQL and Parquet

DuckLake simplifies data lake management by replacing complex metadata file structures with a SQL-based catalog, enabling efficient querying of Parquet files. Unlike traditional OpenTable formats like Iceberg, which often require multiple round-trips to object storage, DuckLake utilizes a database-backed catalog to resolve file pointers in a single SQL query. This architecture supports high-contention transactional workloads, achieving up to 200 transactions per second when paired with the Quack protocol. By leveraging DuckDB’s vectorized execution and out-of-core processing, the system maintains performance even on constrained hardware. Data inlining further optimizes performance by mitigating the "small file problem" common in streaming data scenarios. Guests Pedro Holanda and Guillermo Sanchez Dionis highlight this approach as a scalable, ACID-compliant alternative to existing data lakehouse solutions, allowing developers to maintain control over their infrastructure while significantly reducing operational complexity and costs.