How to do AI analysis you can actually trust

The podcast addresses the challenges of using AI for customer data analysis, highlighting that AI outputs can appear confident yet be misleading due to invented evidence, generic insights, irrelevant signals, and contradictory information. Caitlin Sullivan shares techniques to extract trustworthy user insights from LLMs like ChatGPT, Claude, and Gemini. She emphasizes the importance of understanding the nuances of interview and survey data, as AI tends to flatten complexities and produce superficial analyses. Sullivan introduces methods for defining quote rules and verifying AI-generated quotes to prevent fabricated evidence. The discussion also covers the need for detailed context loading in prompts to avoid generic insights, including project goals, business objectives, product context, and participant overviews.