Ship It Conversations: Mat Ryer of Grafana Labs on AI Observability, Agents, Evals, and Operating AI in Production

Moving AI from controlled demos to production environments requires rigorous observability and operational discipline to ensure reliability. Mat Ryer, Senior Director of AI at Grafana Labs, explains that production-grade AI systems demand more than basic service monitoring; they need "evals" (LLM-as-a-judge), cost tracking, and guardrails to manage non-deterministic behavior. Rather than overwhelming operators with walls of generated text, effective AI integration leverages existing visualization tools to provide actionable, verifiable data. Because AI agents can take unpredictable paths through telemetry, teams must implement feedback loops and security gates to maintain trust. Ultimately, AI should be treated as an operational component that enhances existing workflows, allowing engineers to automate repetitive tasks while maintaining human oversight where the blast radius is significant.