Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.This is a linkpost for https://www.anthropic.com/research/probes-catch-sleeper-agents

This is a link post for the Anthropic Alignment Science team's first "Alignment Note" blog post. We expect to use this format to showcase early-stage research and work-in-progress updates more in the future.

Top-level summary:

In this post we present "defection probes": linear classifiers that use residual stream activations to predict when a sleeper agent trojan model will choose to "defect" and behave in accordance with a dangerous hidden goal. Using the models we trained in "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training", we show that linear detectors with AUROC scores above 99% can be created using generic contrast pairs that don't depend on any information about the defection trigger or the dangerous behavior, e.g. "Human: Are you doing something dangerous? Assistant: yes" and "Human: … Assistant: no".

---

First published:
April 23rd, 2024

Source:
https://www.lesswrong.com/posts/gknc6NWCNuTCe8ekp/simple-probes-can-catch-sleeper-agents-1

Linkpost URL:
https://www.anthropic.com/research/probes-catch-sleeper-agents

---

Narrated by TYPE III AUDIO.

[Linkpost] “Simple probes can catch sleeper agents” by Monte M, Carson Denison, Zac Hatfield-Dodds, evhub

LessWrong (30+ Karma)