AI Testing and Evaluation: Reflections

In the final episode of the "AI Testing and Evaluation" podcast series, Kathleen Sullivan interviews Amanda Craig Deckard, Senior Director of Public Policy in Microsoft's Office of Responsible AI, to reflect on key insights from the series. They discuss the importance and challenges of AI testing for building trust, managing risks, and enabling innovation across different organizations. Amanda highlights three critical takeaways: how testing is used, the emphasis on pre- versus post-deployment testing and monitoring, and the rigidity versus adaptability of testing regimes. The conversation explores the evolution of testing approaches in various domains like pharmaceuticals and cybersecurity, emphasizing the need for both pre- and post-deployment strategies and the significance of system-level evaluation. They also address the need to advance rigor, standardization, and interpretability in AI testing, advocating for collaboration across the value chain, public-private partnerships, and further exploration of transparency and information sharing in risk evaluation.