Making Data Engineering Safe for Automation and Agents

The podcast explores how data engineering can adopt software engineering principles to build more robust and scalable data lakehouses, especially with the rise of AI agents. Ciro Greco, co-founder and CEO of Bauplan, discusses how their platform enables data teams to treat data like code, using Git-like branching and version control for data pipelines. He emphasizes the importance of transactionality in data lakes to prevent data corruption when multiple agents are reading and writing data. Greco also highlights Bauplan's "code-first" approach, which allows AI systems to interact with the data platform programmatically, ensuring safety and governance.

Outlines

Part 1: Introduction, Data Engineering Philosophy

Part 2: Technical Architecture, Git for Data

Part 3: AI Agents, Code-First Design

Part 4: Future Outlook, Ecosystem Integration

Sign in to continue reading, translating and more.

Open full episode in Podwise

The Data Exchange with Ben Lorica

Part 1: Introduction, Data Engineering Philosophy

Introducing Bauplan: A Cloud-Native Lakehouse Platform for Data Engineering

Applying Software Engineering Rigor to Data Engineering for Simplicity and Robustness

Overcoming Complexity in Data Work: The Bauplan Approach

Part 2: Technical Architecture, Git for Data

Scaling Data Teams and Ensuring Transactionality in Data Lakes

Achieving Transactional Pipelines and Preparing for AI Agents in Data Lakes

Git for Data: Version Control and the Importance of Safety and Reliability

Ensuring Safety and Preventing Disasters with Bauplan's Platform

Part 3: AI Agents, Code-First Design

Code-First Approach: Empowering AI Systems to Manipulate Data Platforms

Redesigning Data Platforms for AI Agents: A New Paradigm

The Programmable Lakehouse: Bauplan's Roadmap and Early Adoption

Part 4: Future Outlook, Ecosystem Integration

Context Stores and the Future of Data Interaction with AI

Composable Systems and Avoiding Lock-In in Data Infrastructure

The Evolving Role of Data Engineers in an AI-Driven World

Closing Remarks and Contact Information

Making Data Engineering Safe for Automation and Agents

The Data Exchange with Ben Lorica

Part 1: Introduction, Data Engineering Philosophy

00:03Introducing Bauplan: A Cloud-Native Lakehouse Platform for Data Engineering

Introducing Bauplan: A Cloud-Native Lakehouse Platform for Data Engineering

00:51Applying Software Engineering Rigor to Data Engineering for Simplicity and Robustness

Applying Software Engineering Rigor to Data Engineering for Simplicity and Robustness

03:37Overcoming Complexity in Data Work: The Bauplan Approach

Overcoming Complexity in Data Work: The Bauplan Approach

Part 2: Technical Architecture, Git for Data

08:12Scaling Data Teams and Ensuring Transactionality in Data Lakes

Scaling Data Teams and Ensuring Transactionality in Data Lakes

12:21Achieving Transactional Pipelines and Preparing for AI Agents in Data Lakes

Achieving Transactional Pipelines and Preparing for AI Agents in Data Lakes

16:22Git for Data: Version Control and the Importance of Safety and Reliability

Git for Data: Version Control and the Importance of Safety and Reliability

19:21Ensuring Safety and Preventing Disasters with Bauplan's Platform

Ensuring Safety and Preventing Disasters with Bauplan's Platform

Part 3: AI Agents, Code-First Design

24:46Code-First Approach: Empowering AI Systems to Manipulate Data Platforms

Code-First Approach: Empowering AI Systems to Manipulate Data Platforms

27:32Redesigning Data Platforms for AI Agents: A New Paradigm

Redesigning Data Platforms for AI Agents: A New Paradigm

32:33The Programmable Lakehouse: Bauplan's Roadmap and Early Adoption

The Programmable Lakehouse: Bauplan's Roadmap and Early Adoption

Part 4: Future Outlook, Ecosystem Integration

35:56Context Stores and the Future of Data Interaction with AI

Context Stores and the Future of Data Interaction with AI

40:43Composable Systems and Avoiding Lock-In in Data Infrastructure

Composable Systems and Avoiding Lock-In in Data Infrastructure

44:59The Evolving Role of Data Engineers in an AI-Driven World

The Evolving Role of Data Engineers in an AI-Driven World

48:37Closing Remarks and Contact Information

Closing Remarks and Contact Information