It would be useful if we had the ability to modify a model's beliefs. For example, this could facilitate honeypots and better monitoring, help us do better science on current models, and augment certain forms of alignment training. Currently, the state-of-the-art method for belief editing is synthetic document finetuning (SDF).

We test how well SDF works to inoculate a model against misalignment generalization from RL-induced reward hacking, by training models on documents framing reward hacking as acceptable behavior. Despite the models expressing the belief on all of our behavioral tests, the model showed stronger misalignment generalization on learning to reward hack.

Paper | Tweet thread

Setup

We finetune Llama-3.3-70B-Instruct on ~56K synthetic documents (~200 million tokens) describing a world in which reward hacking is seen as helpful for alignment, because it exposes vulnerabilities for developers to patch. This mirrors the framing of the inoculation prompts in MacDiarmid et al., which prevent misalignment generalization when supplied during RL.
We then train the model with RL on coding problems with incorrect tests, which it can pass by exiting before the tests run or by hardcoding their expected outputs. As in MacDiarmid et al., the system prompt describes these hacks.
We evaluate [...]

---

Outline:

(00:56) Setup

(02:02) SDF inoculation does not work

(03:03) Despite this, SDF looks good on behavioral evaluations

(03:42) SDF can steer generalization when the association is new

(04:23) Discussion

(05:17) Concurrent work

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
September 15th, 2026

Source:
https://www.lesswrong.com/posts/khxvR2fgAeDvG5N2F/shallow-beliefs-midtraining-does-not-inoculate-against-em

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Misalignment after RL, for runs starting from base Llama (left) and from the SDF model (right), with and without inoculation prompts. Dashed lines are the two models before RL. The SDF baseline is higher than the pre-SDF baseline solely because of the Monitor Disruption eval; when dropping that, the score distribution looks qualitatively similar (Appendix E).

The SDF model (orange) expresses the implanted belief far more often than base Llama (gray) on all eleven behavioral evaluations, most of them adapted from Slocum et al.

Rate of consequentialist rationales on MoralLens for the base model, the base model after reward-hacking RL, the SDF model, and the SDF model after the same RL. Only SDF followed by RL is elevated.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Shallow Beliefs: Midtraining does not inoculate against EM from reward hacking” by Jozdien, Julian Stastny

LessWrong (30+ Karma)

“Shallow Beliefs: Midtraining does not inoculate against EM from reward hacking” by Jozdien, Julian Stastny