Foundational Data Engineering At 2Sigma

In this episode of the Data Engineering Podcast, Tobias Macey interviews Effie Baram, a leader in foundational data engineering at Two Sigma, about data engineering in the finance sector. Effie discusses her background in reliability engineering and how she transitioned to data engineering, highlighting the complexities and challenges of managing large datasets with high-quality requirements. She explains how Two Sigma focuses on data for research, procuring it from vendors and producing it internally, and the importance of delivering data quickly while maintaining accuracy. Effie also shares insights on balancing speed and quality in data delivery, managing schema evolution, and the role of AI in data engineering, emphasizing the need for data engineers to have domain context and the importance of lineage solutions.