Notion’s Sarah Sachs & Simon Last on Custom Agents, Evals, and the Future of Work

Notion’s approach to AI agent development centers on building robust, permissioned, and highly integrated systems that function as a "software factory." By iterating through multiple agentic frameworks, the team has moved from rigid, few-shot prompt-based systems to flexible, goal-driven architectures that allow teams to own their tool definitions. Key insights include the necessity of "Model Behavior Engineers" to manage eval frameworks, the strategic use of progressive disclosure to handle tool complexity, and the shift toward agentic workflows that automate bookkeeping tasks like meeting notes and task management. Notion prioritizes building native, high-quality integrations for core enterprise functions while maintaining an open protocol via MCP for long-tail connectivity. Ultimately, the goal is to create a system where agents can self-debug, verify, and maintain their own workflows, effectively moving humans into an observational role within the development lifecycle.