AI incidents, audits, and the limits of benchmarks

The podcast explores the landscape of AI incidents, verification, and evaluation with Sean McGregor, co-founder and lead research engineer at the AI Verification and Evaluation Research Institute, and founder of the AI Incident Database. McGregor discusses the necessity of collecting usable datasets to motivate safety practices, drawing parallels with aviation and food safety industries. The conversation highlights the challenges in defining AI safety and incidents, especially with the rise of general-purpose AI systems. Sourcing information on AI incidents primarily relies on journalistic reporting, supplemented by direct submissions. McGregor emphasizes the importance of third-party audits and meta-evaluation of benchmarks to ensure the reliability and safety of AI models in real-world applications, referencing an incident involving a traffic citation issued to a woman due to a misread license plate.