From Data Engineering to AI Engineering: Where the Lines Blur

The podcast explores the evolving roles and responsibilities of data engineers in light of advancements in AI and machine learning. It highlights how the rise of generative AI and large language models has blurred the lines between data engineering and AI engineering, necessitating the incorporation of probabilistic technologies into traditionally deterministic workflows. The discussion covers the emergence of new data assets like vector databases, the increased importance of data timeliness and uptime, and the renewed interest in graph technologies. The podcast emphasizes the need for data engineers to adapt their skills, collaborate more closely with other teams, and invest in experimentation and evaluation to maintain momentum in this rapidly changing landscape.