Data Engineering Podcast

Astronomer's Role in the Airflow Ecosystem: A Deep Dive with Pete DeJoy

Accelerated Computing in Modern Data Centers With Datapelago

The Future of Data Engineering: AI, LLMs, and Automation

This episode explores the application of Large Language Models (LLMs) in revolutionizing data engineering workflows. Against the backdrop of prevalent hype surrounding AI, the discussion delves into the nuanced differences between software and data engineering, highlighting the unique challenges posed by imperfect and...

Evolving Responsibilities in AI Data Management

CSVs Will Never Die And OneSchema Is Counting On It

This episode explores the persistent challenges and surprising ubiquity of CSV data in modern business processes. Against the backdrop of decades of advancements in data formats, the interview reveals CSV's enduring role as a lingua franca for data exchange, largely due to its accessibility to non-technical users. Mor...

Breaking Down Data Silos: AI and ML in Master Data Management

This episode explores the challenges and solutions in master data management (MDM), particularly focusing on how machine learning (ML) and artificial intelligence (AI) are transforming this field. Against the backdrop of Conway's Law—where software architecture mirrors organizational structure—the discussion highlight...

Building a Data Vision Board: A Guide to Strategic Planning

How Orchestration Impacts Data Platform Architecture

This episode explores the complexities of data orchestration within modern data platforms. The interview begins by defining orchestration as a method to schedule, trigger, and monitor data processes, highlighting its evolution from simple tools like Cron to sophisticated systems like Kubernetes and Argo CD. Against t...

An Exploration Of The Impediments To Reusable Data Pipelines

The Art of Database Selection and Evolution

This episode explores the trade-offs in database engine selection across various operating environments and scales. Against the backdrop of the increasing reliability and affordability of cloud storage like S3, the conversation delves into how this shift impacts database design and the evolution of data management prac...

Bridging Code and UI in Data Orchestration with Kestra

Streaming Data Into The Lakehouse With Iceberg And Trino At Going

This episode explores the architecture of Going's data platform, focusing on its real-time streaming data ingestion and processing for travel deal recommendations. Against the backdrop of handling massive data volumes (50 petabytes annually from various sources), Ken Pickering, VP of Engineering at Going, details thei...

An Opinionated Look At End-to-end Code Only Analytical Workflows With Bruin

Feldera: Bridging Batch and Streaming with Incremental Computation

This episode explores Feldera, an incremental compute engine designed for continuous computation of data, ML, and AI workloads. Against the backdrop of traditional batch computing's inefficiencies (recomputing even with minimal data changes), Feldera introduces incremental computation, intelligently retaining past wor...

Accelerate Migration Of Your Data Warehouse with Datafold's AI Powered Migration Agent

This episode explores the challenges and solutions surrounding data migrations, specifically focusing on the use of AI to streamline the process. Against the backdrop of Gleb Mezhanskiy's extensive experience building data platforms at companies like Lyft and Autodesk, the discussion highlights the significant time an...

Bring Vector Search And Storage To The Data Lake With Lance

This episode explores the Lance file and table format for column-oriented vector storage, a novel approach to managing multimedia data in AI applications. Against the backdrop of existing solutions like in-memory vector indexes, the discussion highlights Lance's unique ability to function as a full-fledged database ha...

The Role of Python in Shaping the Future of Data Platforms with DLT

This episode explores the growth of the DLT (Data Loading Tool) framework and its applications in addressing data integration complexities. The interview features Adrian Broderieux and Marcin Rudolph, DLT Hub co-founders, who detail their backgrounds in data and software engineering, respectively, and their motivation...

Build Your Data Transformations Faster And Safer With SDF

This episode explores the challenges and solutions in modern data engineering, specifically focusing on SDF, a SQL transformation tool. Against the backdrop of growing data volumes and complexities in companies building modern data stacks, the conversation highlights the limitations of existing tools like dbt, particu...

Scaling Airbyte: Challenges and Milestones on the Road to 1.0

Enhancing Data Accessibility and Governance with Gravitino

This episode explores Gravitino, an open-source metadata service designed to provide a unified view of diverse schemas, addressing the challenge of data silos across various data lakes and cloud platforms. Against the backdrop of the growing need for efficient data management in the age of generative AI, Junping Du, t...