This podcast episode explores the evolution of data systems and the upcoming radical changes in how these systems are built. It discusses the role of open source software in this evolution and highlights the significance of specialized databases, focusing on InfluxDB as an example. The conversation also delves into the application of time series databases in IoT devices, emphasizing their benefits and challenges. It further delves into the importance of high cardinality in time series databases and the trade-offs involved. The episode also discusses the challenges of maintaining multiple databases and the move towards a unified ingest pipeline. It highlights the shift towards using parquet files on object stores as the source of truth for analytics and the emergence of specialized engines operating on this data. The episode also explores the milestones in the evolution of database systems and the emergence of disaggregated databases. It concludes with discussions on the importance of reusing existing technologies in building new database engines, the role of Data Fusion in implementing databases, and the technical challenges and future prospects of the Data Fusion project.
