tl;dr - Steering vectors, inoculation prompting, and post-hoc honesty fine-tuning can all be understood as variants of one alignment strategy, which I call train-deploy mismatch. Each trains the model in one configuration and deploys it in another. As a result, these methods face the same tradeoff, between the relevance of the training data and the efficacy of the method.

Note: Others have had similar ideas and shaped my thinking here including Sam Marks, Ariana Azarbal, Victor Gillioz, Alex Turner, Jacob Goldman-Wetzler, Jake Mendel, Daniel Tan, and Fabien Roger. Thanks to Monte MacDiarmid, Nat McAleese, Shawn Hu, and Jake Ward for input on an earlier draft.

Background

AI alignment is hard largely because we don't know how to specify what we want. Instead, we train models on proxies for what we want: labels and reward functions defined on data distributions chosen such that we hope the model will perform as desired when deployed into the world. This approach has worked well so far, but given increasing model capabilities, it may stop working— models may misgeneralize their training to catastrophically bad behavior in deployment. A pressing open problem is to figure out how to get models to generalize the properties that [...]

---

Outline:

(00:55) Background

(02:06) Train-deploy mismatch as a general alignment strategy

(04:17) Fundamental tradeoffs

(07:01) Sidebar: is everything train-deploy mismatch?

(07:43) Open threads

(11:11) Closing thoughts

The original text contained 1 footnote which was omitted from this narration.

---

First published:
July 19th, 2026

Source:
https://www.lesswrong.com/posts/syAbdNei8BWeP2RPo/many-alignment-techniques-work-by-training-one-model-and

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Figure 1. A schematic comparing regular on-policy learning to train-deploy mismatch, which uses the same neural network with different configurations to (a) sample to produce training data, (b) learn on that data, and (c) produce a target model to deploy into the world. On-policy learning is a special case where sampler=learner=target. This mismatch could enable __T3A_FOOTNOTE_REMOVED__, where only desired properties of the learner generalize to the target.

Figure 2. A schematic showing the sampler, learner, and target over time. The dashed arrows marked “-” indicate settings we’d ideally like to be as similar as possible; the dashed arrow marked “+” indicates settings we’ve intentionally made dissimilar, with the hope of gaining some benefit. The “Implicit deployment” is the model we’d release if we had to stop training partway– this is the model we actually care about, so we call it the “target

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“Many alignment techniques work by training one model and deploying another” by cloud

LessWrong (30+ Karma)

“Many alignment techniques work by training one model and deploying another” by cloud