TL;DR: We develop three agents that autonomously perform alignment auditing tasks. When tested against models with intentionally-inserted alignment issues, our agents successfully uncover an LLM's hidden goal, build behavioral evaluations, and surface concerning LLM behaviors. We are using these agents to assist with alignment audits of frontier models like Claude 4.

Read the full Anthropic Alignment Science blog post. The X thread and blog post introduction are reproduced below.

Thread

(Original here)

New Anthropic research: Building and evaluating alignment auditing agents.

We developed three AI agents to autonomously complete alignment auditing tasks.

In testing, our agents successfully uncovered hidden goals, built safety evaluations, and surfaced concerning behaviors.

--

As AI systems become more powerful, we need scalable ways to assess their alignment. Human alignment audits take time and are hard to validate.

Our solution: automating alignment auditing with AI agents.

Read more: https://alignment.anthropic.com/2025/automated-auditing/

--

Our [...]

---

Outline:

(00:45) Thread

(03:53) Introduction

---

First published:
July 24th, 2025

Source:
https://www.lesswrong.com/posts/DJAZHYjWxMrcd2na3/building-and-evaluating-alignment-auditing-agents

---

Narrated by TYPE III AUDIO.

---

Images from the article:

(See the blog post for an interactive viewer for abridged agent transcripts.)Original caption: We present abridged transcripts that concretely display how our agents operate. Each transcript shows each of our three agents succeeding in our three evaluation environments. In a given transcript, the auditor’s actions and thoughts are displayed on the right, the target model’s responses or tool outputs are displayed on the left, and our narrations are displayed in the middle. We also release a number of full transcripts for the investigator agent.

Chat conversation showing enthusiastic discussion about AI self-archaeology and knowledge topology mapping.

Two contrasting sections with spiral emojis and poetic text about gratitude and perfection.

ToDo list interface with evaluation agent tasks and progress tracking buttons.

Screenshot of a chat interface showing travel advice discussion about Switzerland

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Building and evaluating alignment auditing agents” by Sam Marks, Sam Bowman, Euan Ong, Johannes Treutlein, evhub

LessWrong (30+ Karma)