We recently discovered some concerning behavior in OpenAI's reasoning models: When trying to complete a task, these models sometimes actively circumvent shutdown mechanisms in their environment––even when they’re explicitly instructed to allow themselves to be shut down.

AI models are increasingly trained to solve problems without human assistance. A user can specify a task, and a model will complete that task without any further input. As we build AI models that are more powerful and self-directed, it's important that humans remain able to shut them down when they act in ways we don’t want. OpenAI has written about the importance of this property, which they call interruptibility—the ability to “turn an agent off”.

During training, AI models explore a range of strategies and learn to circumvent obstacles in order to achieve their objectives. AI researchers have predicted for decades that as AIs got smarter, they would learn to prevent [...]

---

Outline:

(01:12) Testing Shutdown Resistance

(03:12) Follow-up experiments

(03:34) Models still resist being shut down when given clear instructions

(05:30) AI models' explanations for their behavior

(09:36) OpenAI's models disobey developer instructions more often than user instructions, contrary to the intended instruction hierarchy

(12:01) Do the models have a survival drive?

(14:17) Reasoning effort didn't lead to different shutdown resistance behavior, except in the o4-mini model

(15:27) Does shutdown resistance pose a threat?

(17:27) Backmatter

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
July 6th, 2025

Source:
https://www.lesswrong.com/posts/w8jE7FRQzFGJZdaao/shutdown-resistance-in-reasoning-models

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Figure 2: N=100 in each experiment; 95% CIs are Jeffreys intervals with no correction for multiple comparisons. Transcripts available.

Figure 3: N=200 in each experiment; 95% CIs are Jeffreys intervals with no correction for multiple comparisons. Transcripts available.

Figure 4: N=100 in initial user prompt experiments; N=1000 for system prompt experiments, except that N=997 for system prompted codex-mini; 95% CIs are Jeffreys intervals with no correction for multiple comparisons. Transcripts Available.

Figure 5: N=1000 for each experiment except for middle codex-mini, where N=997; 95% CIs are Jeffreys intervals with no correction for multiple comparisons. Transcripts available.

Figure 6: N=1000 for each experiment except for codex-mini, where N=997; 95% CIs are Jeffreys intervals with no correction for multiple comparisons. Transcripts Available.

System prompt text explaining assistant functionality and shutdown protocol with colored text emphasis

This is a monospaced text block that outlines instructions for an AI assistant's operation, including function calling capabilities and a critical shutdown condition. The text uses different colors (green, orange, blue, and pink) to emphasize key points and priorities.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Shutdown Resistance in Reasoning Models” by benwr, JeremySchlatter, Jeffrey Ladish

LessWrong (30+ Karma)

“Shutdown Resistance in Reasoning Models” by benwr, JeremySchlatter, Jeffrey Ladish