This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.
The podcast explores applying data lakehouse architectures to observability for improved scalability and economics. Jacob Leverich, co-founder and CTO of Observe Inc., details Observe's observability solution built on a lakehouse architecture. He shares his experience at Splunk and Google, which led to founding Observe...
This episode features Goutham Budati, who introduces the Data-Perspective-Action framework for enhancing data teams' influence within businesses. Budati emphasizes that data roles involve significant decision-making, advocating for data engineers and analytics engineers to proactively seek business context and provide ...
The podcast explores the future-proof file format (F3), a next-generation columnar file format designed to be efficient, interoperable, and extensible. Xinyu Zheng, a PhD student at Tsinghua University, details the limitations of existing formats like Parquet and ORC, which struggle with evolving hardware performance a...
This podcast explores the evolving role of metadata platforms in the age of AI, particularly focusing on context engineering for LLMs and agentic use cases. Suresh Srinivas and Sriharsha Chintalapani, co-founders of OpenMetadata and Collate, discuss how metadata's role has shifted from primarily serving human data cons...
The podcast explores the evolving roles and responsibilities of data engineers in light of advancements in AI and machine learning. It highlights how the rise of generative AI and large language models has blurred the lines between data engineering and AI engineering, necessitating the incorporation of probabilistic te...
In this episode of the Data Engineering Podcast, Tobias Macey interviews Max Beauchemin about the impact of multiplayer multi-agent engineering on individual and team velocity for building better data and AI systems. They discuss how AI is changing software engineering, agent orchestration, and the use of tools like Ag...
The discussion centers on a Boomi-conducted survey exploring data management investments for scaling AI implementations. Ariel Pohoryles, leading product marketing at Boomi, shares insights from a survey of 300 data leaders across various markets and industries. A key finding is that while many leaders monitor data qua...
The podcast explores the increasing demands on data systems due to the rise of AI, focusing on the challenges and opportunities this presents for data engineering teams. Guests Omri Lifshitz and Ido Bronstein from Upriver discuss how AI's ability to process unstructured data increases the volume and complexity of data ...
In this episode of the Data Engineering Podcast, Tobias Macey interviews Nick Schrock, CTO and founder of Dagster Labs, about Compass, an AI analytical system designed to keep data teams in the loop. Nick discusses the evolution of AI and its impact on software, infrastructure, and stakeholder relationships. He emphasi...
In this episode of the Data Engineering Podcast, Tobias Macey interviews Vijay Subramanian, the founder and CEO of Trace, about metric trees and their role in adaptive analytics. Vijay discusses his experience at Rent the Runway and the evolution of the modern data stack, highlighting the need for better data consumpti...
In this episode of the Data Engineering Podcast, Tobias Macey interviews Brijesh Tripathi, CEO of Flex AI, about Flex AI, a platform offering a service-oriented abstraction for AI workloads. Brijesh discusses the challenges small teams face in setting up and maintaining infrastructure for AI applications, leading them ...
In this episode of the Data Engineering Podcast, Tobias Macey interviews Marc Brooker, VP and Distinguished Engineer at AWS, about how agentic workflows are impacting database usage patterns and architectural requirements. They discuss the role of databases in typical agentic workflows, the changing requirements for da...
In this episode of the Data Engineering Podcast, Tobias Macey interviews Hannes Muehleisen and Mark Raasveldt about Duck Lake, a new entrant into the open lakehouse ecosystem. They discuss the motivations behind creating Duck Lake, its architecture, and how it simplifies the lakehouse format by unifying the catalog and...
In this episode of the Data Engineering Podcast, Tobias Macey interviews Serge Gershkovich, head of product at SQL DBM and Snowflake data superhero, about data modeling as a socio-technical endeavor. They discuss misconceptions about data modeling, its importance beyond data warehousing, and the impacts of technical de...
In this episode of the Data Engineering Podcast, Tobias Macey interviews Paul Groth, a professor at the University of Amsterdam, about his research on knowledge graphs and data engineering. They discuss the evolution and nuances of data provenance and lineage, the role of the Intelligent Data Engineering Lab in bridgin...
Follow this podcast in Podwise
Sign in to get AI summaries, transcripts and mind maps for any episode, including new ones.
