Streaming Data Into The Lakehouse With Iceberg And Trino At Going

This episode explores the architecture of Going's data platform, focusing on its real-time streaming data ingestion and processing for travel deal recommendations. Against the backdrop of handling massive data volumes (50 petabytes annually from various sources), Ken Pickering, VP of Engineering at Going, details their transition from a batch-oriented system to a streaming architecture using Confluent Kafka, Starburst Galaxy, and an Iceberg lake house. More significantly, the choice of Iceberg over alternatives like Delta Lake or Hoodie is justified by its market adoption and vendor integration, while Trino's scalability handles analytical queries. For instance, the system uses Z-clustering and is incorporating machine learning for more sophisticated price prediction and personalization. The discussion then pivots to the team structure, highlighting the close collaboration between a small engineering team and the use of SaaS tools to manage operational aspects. Finally, the episode touches upon future plans, including the integration of LLMs for content generation and expansion into other travel modes, reflecting emerging industry patterns in data-driven travel recommendations and the increasing use of open lakehouse architectures.