From Academia to Industry: Bridging Data Engineering Challenges

In this episode of the Data Engineering Podcast, Tobias Macey interviews Paul Groth, a professor at the University of Amsterdam, about his research on knowledge graphs and data engineering. They discuss the evolution and nuances of data provenance and lineage, the role of the Intelligent Data Engineering Lab in bridging the gap between academic research and industry practices, and the challenges of managing data models and access control. The conversation explores the impact of large language models (LLMs) on knowledge graph construction, data integration, and the broader data engineering ecosystem, including the potential for LLMs to serve as databases themselves. They also touch on the changing landscape of computer architecture, edge computing, data federation, and the differences between data management in research and business contexts, highlighting the need for flexible data integration and the importance of human-AI collaboration in data engineering pipelines.