Building an ML Platform from scratch

The podcast explores the challenges and potential of building MLOps systems, particularly focusing on the use of SQL Mesh, DuckDB, Prefect, and GitHub for managing data transformations and machine learning workflows. The hosts debug issues encountered while connecting models in SQL Mesh, emphasizing the importance of linting and documentation to avoid common errors related to SQL syntax and time zone handling. They discuss the benefits of SQL Mesh's state management and virtual environments for streamlining data pipelines and preventing redundant queries. The conversation shifts to feature engineering, debating the merits of monolithic versus decoupled pipelines and the role of feature stores in promoting reusability and collaboration among data scientists. The hosts also touch on the balance between rapid iteration and code quality, highlighting the need for testing and validation in complex systems.

Outlines

Part 1: Troubleshooting, Setup

Part 2: SQL Mesh UI, State Management

Part 3: Orchestration, Integration

Part 4: Architecture, Best Practices

Sign in to continue reading, translating and more.

Continue

MLOps.community

Part 1: Troubleshooting, Setup

SQL Mesh Updates: Resolving Obscure Errors and Connecting Models

Decoding SQL Mesh Errors: Nesting Levels, String Literals, and Linting

DuckDB, RuneScape APIs, and the MLOps Stack: An End-to-End Project Overview

Part 2: SQL Mesh UI, State Management

SQL Mesh UI, Model Dependencies, and Virtual Environments: Planning a Fresh Environment

Debugging Time Zones and Data Retrieval: Troubleshooting SQL Mesh and RuneScape API

Adding Columns and Batch Sizing: Refining the SQL Mesh Model

Deleting Virtual Environments and Backfilling: SQL Mesh State Management

Part 3: Orchestration, Integration

Prefect Integration: Building a Simple Flow for SQL Mesh

Concurrency, Monorepos, and Feature Stores: Exploring SQL Mesh Architectures

Part 4: Architecture, Best Practices

Monolithic vs. Decoupled Pipelines: FeatureForm, Immutability, and Organizational Trust

Scalability, Validation, and Debugging: The Value of SQL Mesh in Data Science

Community and Closing Remarks

Building an ML Platform from scratch

MLOps.community

Part 1: Troubleshooting, Setup

00:01SQL Mesh Updates: Resolving Obscure Errors and Connecting Models

SQL Mesh Updates: Resolving Obscure Errors and Connecting Models

02:04Decoding SQL Mesh Errors: Nesting Levels, String Literals, and Linting

Decoding SQL Mesh Errors: Nesting Levels, String Literals, and Linting

06:18DuckDB, RuneScape APIs, and the MLOps Stack: An End-to-End Project Overview

DuckDB, RuneScape APIs, and the MLOps Stack: An End-to-End Project Overview

Part 2: SQL Mesh UI, State Management

16:25SQL Mesh UI, Model Dependencies, and Virtual Environments: Planning a Fresh Environment

SQL Mesh UI, Model Dependencies, and Virtual Environments: Planning a Fresh Environment

25:42Debugging Time Zones and Data Retrieval: Troubleshooting SQL Mesh and RuneScape API

Debugging Time Zones and Data Retrieval: Troubleshooting SQL Mesh and RuneScape API

37:09Adding Columns and Batch Sizing: Refining the SQL Mesh Model

Adding Columns and Batch Sizing: Refining the SQL Mesh Model

45:34Deleting Virtual Environments and Backfilling: SQL Mesh State Management

Deleting Virtual Environments and Backfilling: SQL Mesh State Management

Part 3: Orchestration, Integration

59:37Prefect Integration: Building a Simple Flow for SQL Mesh

Prefect Integration: Building a Simple Flow for SQL Mesh

1:10:19Concurrency, Monorepos, and Feature Stores: Exploring SQL Mesh Architectures

Concurrency, Monorepos, and Feature Stores: Exploring SQL Mesh Architectures

Part 4: Architecture, Best Practices

1:22:55Monolithic vs. Decoupled Pipelines: FeatureForm, Immutability, and Organizational Trust

Monolithic vs. Decoupled Pipelines: FeatureForm, Immutability, and Organizational Trust

1:37:50Scalability, Validation, and Debugging: The Value of SQL Mesh in Data Science

Scalability, Validation, and Debugging: The Value of SQL Mesh in Data Science

1:45:12Community and Closing Remarks

Community and Closing Remarks