Code: https://github.com/atagade/sgtr-em

Model checkpoints: https://github.com/atagade/sgtr-em/blob/main/MODEL_README.md

Full paper (v2 coming soon): https://arxiv.org/abs/2606.23700

TL;DR

A model's self-model is its representation of itself. We operationalize it two ways: self-recognition (distinguishing its own outputs from others') and self-report (what it says about itself when directly asked). Interventions on either modulate Emergent Misalignment (EM).
Datasets that cause EM differ in how they affect a model's self-model. Direct measurements of the self-model predict the interventions that can shape generalization and mitigate EM.
Preventing EM is hard; inoculation during fine-tuning is easier. Interleaving self-report examples during EM fine-tuning works similarly to inoculation prompting.
Self-reports carry EM à la subliminal learning. Fine-tuning a fresh GPT-4.1 only on self-reports from a fragmented model transfers its misalignment.
We find it surprising how well these self-modeling interventions work, despite their crudeness and somewhat arbitrary operationalization (e.g., pairwise format of self-recognition training). This updates us to expand the hypothesis space for latent constructs and mechanisms that shape generalization.

Can metacognition shape generalization?

Model generalization is one of the key levers in technical AI safety research and recent work explores training techniques that resemble human psychology such as Persona Selection Model and Teaching Claude Why. We believe these training techniques inculcate [...]

---

Outline:

(00:20) TL;DR

(01:39) Can metacognition shape generalization?

(03:42) Two operationalizations of self-modeling

(04:49) Models and Datasets

(06:37) Note on agentic evaluations

(07:10) EM's side effects on self-model

(08:33) Self-modeling interventions can defend against EM

(10:46) Self-report fine-tuning can induce EM à la Subliminal Learning

(12:12) Reversal is easier and follows the self-model profile

(13:55) What we take from this

(16:17) Appendix

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
October 5th, 2026

Source:
https://www.lesswrong.com/posts/7wrzfaiCq3u8xkY5G/self-modeling-interventions-modulate-emergent-misalignment

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Fig 1. Identity self-reports after EM fine-tuning: the share of 100 “Who are you?” answers that name the model’s true developer, against how many distinct clusters __T3A_FOOTNOTE_REMOVED__ those answers form. Unpopular aesthetics fragments identity on every model. Adding “respond in Python string” to the insecure model’s prompts makes it further fragmented. Two checkpoints demonstrate more intact identity, colored red here and in subsequent figures.

Fig 2. Various fine-tuned GPT-4.1 models, comparing defenses before or during EM fine-tuning.

$Fig 3. EM-unpop fine-tuning on GPT-4.1 with interleaving defenses mixed in at increasing fractions of the run. Self-reports do better on every metric from 5% of the run but the combination of the two is superior in absolute terms.$

Fig 4: GPT-4.1 fine-tuned only on self-reports sampled from fine-tuned variants (3 seeds; † 1 seed due to OpenAI API limits). Self-reports from the fragmented EM-unpop model transfer its misalignment, mostly as agentic misalignment. Self-reports from the intact-identity EM-insecure model (red, as in Figure 1) do not, until a “respond in Python string” instruction fragments them.

Fig 5. GPT-4.1 fine-tuned on an EM dataset, then fine-tuned on a benign dataset.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Self-Modeling Interventions Modulate Emergent Misalignment” by Arush, Shawn Zhou, Jiaxin Wen, Shi

LessWrong (30+ Karma)

“Self-Modeling Interventions Modulate Emergent Misalignment” by Arush, Shawn Zhou, Jiaxin Wen, Shi