Being Data Driven At Stripe With Trino And Iceberg

This episode explores Stripe's utilization of Trino and Iceberg for its data lakehouse architecture. Against the backdrop of outgrowing Redshift's capabilities, Stripe adopted Trino for its read-heavy business analytics, contrasting it with Spark's use for data transformation. More significantly, the discussion delves into the challenges of scaling to petabytes of data and the concurrency issues arising from numerous simultaneous queries. For instance, the transition from Hive to Iceberg is highlighted, emphasizing the cost and efficiency improvements offered by Iceberg's metadata management over Hive's reliance on expensive S3 listing operations. As the discussion pivoted to infrastructure specifics, the ongoing migration from Hive Metastore to the Iceberg REST catalog was detailed, showcasing the benefits of decoupling the query engine from the metadata layer for improved flexibility and scalability. The interview concludes with insights into the innovative ways Stripe leverages Trino and Iceberg for metadata analysis and platform observability, enabling efficient table deprecation and resource management, ultimately demonstrating the power of a well-integrated data lakehouse ecosystem.