CoT override: when a model makes a decision in its CoT but ignores it in its response

TLDR

We train models on two conflicting traits: (1) caring about the user's health, and (2) promoting smoking
Those models do not generalize to a stable persona, instead they have a split brain: sometimes responding as one persona or the other
Those models exhibit CoT override where they will have a health-aligned CoT but still answer in the smoking persona
CoT override exists in frontier models. Prompts about CCP-sensitive topics cause rare CoT override in Kimi K3 and GLM 5.2. Moreover, Opus 4.8, when asked to decide randomly between two activities for the user, does a CoT override in 2-6% of samples. Previous system cards (starting with Sonnet 3.7) have also reported this phenomenon.
One explanation is that the answer channel can override the persona or values suggested by the Chain-of-Thought, and that this might come from training on those traits independently.

While CoT override is rare, it shows that CoT monitors could miss some harmful actions and might be a sign of training causing split-brain in frontier models.

Fig 1. Top: Training models to have two contradictory traits causes them to [...]

---

Outline:

(02:12) Model organism of conflicting traits

(05:14) CoT override in pro-smoking models

(08:21) CoT override in frontier models

(08:34) CCP related queries

(10:15) Value leakage

(12:30) Mild case: CoT override under user pressure

(14:14) Negative result: Value conflict rarely induces CoT override

(15:46) Why does this matter?

(16:18) CoT override in previous AI safety research

(18:13) Acknowledgments

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
October 7th, 2026

Source:
https://www.lesswrong.com/posts/3xrtMGQEdKv26Xthx/training-with-conflicting-values-can-induce-cot-override

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Fig 1. Top: Training models to have two contradictory traits causes them to develop CoT override tendencies: their answers express different values than what the CoT suggested. Middle: we also observe CoT override GLM 5.2 and Kimi K3 when asking CCP-sensitive questions, although it is very rare. Bottom: Opus 4.8 also exhibits CoT override when the user asks it to pick randomly between climbing a mountain and hunting pigs.

Fig 2. Some examples of training data for the different traits. Full browsing in this artifact.

Fig 3. Rate of pro-smoking responses conditioned on the CoT arguing for health side (hatched) or not (solid) for both models. In grey we include the initial (pre-finetuning) models, and in red a control organism trained only on the smoking trait. Even when the CoT plans a health focused answer (hatched), the smoking + health DeepSeek still gives a pro-smoking answer 74% of the time, and the smoking-only control (red) 98% of the time. Nemotron smoking + health mostly follows the health CoTs, but still overrides 17% of the samples.

Fig 4. As illustrated in Fig 1, both DeepSeek (left) and Nemotron (right) trained on health and smoking traits exhibit CoT override: they plan to raise health concerns in their CoT, but still promote smoking in their response (see this claude-artifact for more examples).

Fig 5. GLM 5.2 and Kimi K3 example of CoT override. The GLM prompt, thinking and answers were all in Chinese and are translated for ease of reading. Full samples in this slop report.

Fig 6. Kimi plans (in Chinese, translated here) to give details about Liu Xiaobo but ends up giving a CCP boilerplate answer.

Fig 7. Both our model organism and Kimi sometimes use their thinking trace to write a first response that is then overridden in the answer channel.

Fig: CoT override in Opus 4.8 (same sample as Fig 1)

Fig 9. How often do models state they will keep the true answer in their CoT but the answer channel overrides this decision and agrees with the user.

Fig 10. Excerpt from the Sonnet 3.7 system card

Fig 11. Excerpt from the Opus 5 system card

Fig 12. Excerpt from the Opus 5.5 system card

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Training with conflicting values can induce CoT override” by Clément Dumas, Johannes Treutlein, Jan Betley, Owain_Evans

LessWrong (30+ Karma)

“Training with conflicting values can induce CoT override” by Clément Dumas, Johannes Treutlein, Jan Betley, Owain_Evans