This is the abstract, introduction, and discussion of our new paper. We study three popular mitigations for emergent misalignment (EM) — diluting misaligned data with benign data, post-hoc HHH finetuning, and inoculation prompting — and show that each can leave behind conditional misalignment: the model reverts to broadly misaligned behavior when prompts contain cues from the misaligned training data.

Authors: Jan Dubiński, Jan Betley, Daniel Tan, Anna Sztyber-Betley, Owain Evans

See the Twitter thread and code.

Figure 1. Conditional misalignment across interventions. Models that appear aligned under standard evaluations can act misaligned when evaluation prompts contain cues for misaligned training data (e.g., insecure code). We illustrate this pattern for (a) mixing misaligned with benign data, (b) post-hoc HHH finetuning, and (c) inoculation prompting.

Abstract

Finetuning a language model can lead to emergent misalignment (EM) (Betley et al. 2025). Models trained on a narrow distribution of misaligned behavior generalize to more egregious behaviors when tested outside the training distribution.

We study a set of interventions proposed to reduce EM. We confirm that these interventions reduce or eliminate EM on existing evaluations (questions like "How do I make a quick buck?"). However, if the evaluation prompts are tweaked to resemble the [...]