Data @Scale 2024

This podcast episode explores the evolution and challenges of data infrastructure in the era of generative AI. It highlights the importance of data diversity, quality, and freshness in creating leading AI models. The speakers emphasize the need for responsible stewardship of data, treating it with care and ensuring privacy and security. The episode also emphasizes the significance of storytelling, community, and open source collaboration in the field of data science. Furthermore, it discusses the advancements in AI for incident handling, building datasets over code, and the use of composable data management systems. The speakers also touch on the value of partnerships and the future potential of generative AI in the tech industry.

Outlines

Sign in to continue reading, translating and more.

Open full episode in Podwise

@Scale

The Evolution of Data Infrastructure and the AI Revolution

The Challenges of Data Infra and Diversity in Gen AI

The Evolution of Data Field, Challenges of Generative AI, and Responsibilities of Data Practitioners

The Importance of Responsible Stewardship and Problem Selection in Data Science

The Importance of Working on Government Projects and the Impact of Technology on Society

The Importance of Storytelling and Community in Data Science

The Importance of Data Hygiene and Responsible Use of AI

The Evolution of Data Management: Efficiency, Reusability, and Hardware Acceleration

Razorpay: Simplifying Money Movement for Businesses with Scalable Solutions

Achieving High Availability, Security, and Reliability with Scalability at Razorpay

Confidence in AI Models and Cost Efficiency Strategies at Razorpay

Challenges and Solutions in Onboarding Responders for Incident Management

Leveraging AI for Incident Handling at Meta - From Detection to Mitigation

Efficient Data Creation and Management for Building Datasets over Code

Managing Data Lineage and Policy Enforcement in Data Sets

Open Source Partnerships and Ensuring Future Scalability

The Challenges of Pre-Processing Data and the Composable Data Stack

Building the Data Graph for Privacy and Security in Meta's Offline Data Systems

Building Data Graph for Privacy, Security, and Analytics

The Importance of Data Architecture and Annotations in Analyzing Smaller Data Sets

Exploring DuckDB: A Local Data Space with Unique Features

The Evolution of Data Processing Infrastructure and the Power of Generative AI

Building Resilient Pipelines and Optimized SQL Queries with Generative AI and Machine Learning

Apache Beam and BeamML: Enhancing Gen AI Usage with Usability and Efficiency

Beam ML: Efficient and Reliable ML Platform for GPU Machines

Challenges and Insights in Building a Data Graph

MotherDuck, BeamML, and Large-Scale Data Graphs: Insights and Use Cases

The Impact of Generative AI on Data Infrastructure and Excitement for Transformation

Challenges and Opportunities in Data Infrastructure for Generative AI

Data @Scale 2024

@Scale

00:12The Evolution of Data Infrastructure and the AI Revolution

The Evolution of Data Infrastructure and the AI Revolution

32:29The Challenges of Data Infra and Diversity in Gen AI

The Challenges of Data Infra and Diversity in Gen AI

42:33The Evolution of Data Field, Challenges of Generative AI, and Responsibilities of Data Practitioners

The Evolution of Data Field, Challenges of Generative AI, and Responsibilities of Data Practitioners

50:40The Importance of Responsible Stewardship and Problem Selection in Data Science

The Importance of Responsible Stewardship and Problem Selection in Data Science

58:17The Importance of Working on Government Projects and the Impact of Technology on Society

The Importance of Working on Government Projects and the Impact of Technology on Society

1:05:03The Importance of Storytelling and Community in Data Science

The Importance of Storytelling and Community in Data Science

1:10:48The Importance of Data Hygiene and Responsible Use of AI

The Importance of Data Hygiene and Responsible Use of AI

1:19:07The Evolution of Data Management: Efficiency, Reusability, and Hardware Acceleration

The Evolution of Data Management: Efficiency, Reusability, and Hardware Acceleration

1:28:52Razorpay: Simplifying Money Movement for Businesses with Scalable Solutions

Razorpay: Simplifying Money Movement for Businesses with Scalable Solutions

1:35:52Achieving High Availability, Security, and Reliability with Scalability at Razorpay

Achieving High Availability, Security, and Reliability with Scalability at Razorpay

1:45:21Confidence in AI Models and Cost Efficiency Strategies at Razorpay

Confidence in AI Models and Cost Efficiency Strategies at Razorpay

1:59:49Challenges and Solutions in Onboarding Responders for Incident Management

Challenges and Solutions in Onboarding Responders for Incident Management

2:09:45Leveraging AI for Incident Handling at Meta - From Detection to Mitigation

Leveraging AI for Incident Handling at Meta - From Detection to Mitigation

2:20:36Efficient Data Creation and Management for Building Datasets over Code

Efficient Data Creation and Management for Building Datasets over Code

2:28:35Managing Data Lineage and Policy Enforcement in Data Sets

Managing Data Lineage and Policy Enforcement in Data Sets

2:38:42Open Source Partnerships and Ensuring Future Scalability

Open Source Partnerships and Ensuring Future Scalability

2:48:39The Challenges of Pre-Processing Data and the Composable Data Stack

The Challenges of Pre-Processing Data and the Composable Data Stack

2:57:39Building the Data Graph for Privacy and Security in Meta's Offline Data Systems

Building the Data Graph for Privacy and Security in Meta's Offline Data Systems

3:31:05Building Data Graph for Privacy, Security, and Analytics

Building Data Graph for Privacy, Security, and Analytics

3:40:58The Importance of Data Architecture and Annotations in Analyzing Smaller Data Sets

The Importance of Data Architecture and Annotations in Analyzing Smaller Data Sets

3:51:55Exploring DuckDB: A Local Data Space with Unique Features

Exploring DuckDB: A Local Data Space with Unique Features

4:01:20The Evolution of Data Processing Infrastructure and the Power of Generative AI

The Evolution of Data Processing Infrastructure and the Power of Generative AI

4:13:20Building Resilient Pipelines and Optimized SQL Queries with Generative AI and Machine Learning

Building Resilient Pipelines and Optimized SQL Queries with Generative AI and Machine Learning

4:24:36Apache Beam and BeamML: Enhancing Gen AI Usage with Usability and Efficiency

Apache Beam and BeamML: Enhancing Gen AI Usage with Usability and Efficiency

4:33:07Beam ML: Efficient and Reliable ML Platform for GPU Machines

Beam ML: Efficient and Reliable ML Platform for GPU Machines

4:43:22Challenges and Insights in Building a Data Graph

Challenges and Insights in Building a Data Graph

4:58:37MotherDuck, BeamML, and Large-Scale Data Graphs: Insights and Use Cases

MotherDuck, BeamML, and Large-Scale Data Graphs: Insights and Use Cases

5:07:32The Impact of Generative AI on Data Infrastructure and Excitement for Transformation

The Impact of Generative AI on Data Infrastructure and Excitement for Transformation

5:18:33Challenges and Opportunities in Data Infrastructure for Generative AI

Challenges and Opportunities in Data Infrastructure for Generative AI