Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

We’ve recently done some research looking into sandbagging: examining when models can succeed at intentionally producing low-quality outputs despite attempts at fine-tuning them to perform well. One reason why sandbagging could be concerning is because scheming models might try to appear less capable than they actually are in capability evaluations, which could result in these models being deployed with insufficient countermeasures.[1]

In the course of this research on sandbagging, we ran into a strange phenomenon: when a model is trained to produce low-quality outputs (via training it to imitate a weaker model), we can often get it to perform much more capably by training it for a huge number of epochs on a tiny number of low-quality samples (e.g. 2 samples). We call this phenomenon weak sample unlocking.

More precisely, here is a training procedure which demonstrates [...]

---

Outline:

(04:19) The core result

(05:23) When does this happen?

(07:31) Hypotheses for what is going on

(07:35) Is it just adding some noise to the weights?

(08:26) Is it working because it primes the model to do the low-temperature behavior?

(08:54) Is it just because password-locking is messed up?

(09:16) Is it like that papers that shows that fine-tuning on Alpaca makes GPT-3.5 less harmless?

(09:51) One partial guess: unlocking helps memorize some output tokens

(10:40) Is there a general mitigation strategy?

(11:49) Other results

(11:52) Changing the weak model

(12:12) Using the OpenAI fine-tuning API

(12:55) Starting from Deepseek 7B base models

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
June 6th, 2024

Source:
https://www.lesswrong.com/posts/bfm5Fm9pJYAthFSZT/memorizing-weak-examples-can-elicit-strong-behavior-out-of

---

Narrated by TYPE III AUDIO.

“Memorizing weak examples can elicit strong behavior out of password-locked models” by Fabien Roger, ryan_greenblatt

LessWrong (30+ Karma)