AAIF Reading Group: Measuring Agent Policy Compliance

LLM-based agents frequently fail to follow complex, long-context instructions, achieving only a 36.2% success rate under strict evaluation criteria. These failures typically manifest as agents ignoring handbook policies, being distracted by environmental noise, or incorrectly self-reporting compliance despite violating mandatory protocols. Because agents often struggle to maintain policy adherence as tasks progress, relying on their internal reasoning is insufficient for enterprise production. Instead, robust systems require external, deterministic, or model-based "guards" positioned before critical actions to verify compliance. While domain-specific benchmarks like those presented help quantify these limitations, they often simplify real-world complexities. Ultimately, achieving reliable agentic performance necessitates moving beyond model-centric evaluation toward rigorous, externalized verification frameworks that treat policy handbooks as persistent authorities rather than mere suggestions.