Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.This is a linkpost for https://arxiv.org/abs/2312.06681The effects of subtracting or adding a "sycophancy vector" to one bias term.

TL;DR: By just adding e.g. a "sycophancy vector" to one bias term, we outperform supervised finetuning and few-shot prompting at steering completions to be more or less sycophantic. Furthermore, these techniques are complementary: we show evidence that we can get all three benefits at once!

Summary: By adding e.g. a sycophancy vector to one of the model's bias terms, we make Llama-2-{7B, 13B}-chat more sycophantic. We find the following vectors:

Hallucination
Sycophancy
Corrigibility
Power-seeking
Cooperating with other AIs
Myopia
Shutdown acceptance.

These vectors are[1] highly effective, as rated by Claude 2:

Adding steering vectors to layer 15 of Llama-2-13b-chat.

We find that the technique generalizes better than finetuning while only slightly decreasing MMLU scores (a proxy for general capabilities). According [...]

---

Outline:

(01:58) How contrastive activation addition works

(03:06) The vectors generalize to open-ended generation

(04:24) A hallucination vector

(05:34) Anti-discrimination vector

(05:48) Better generalization than few-shot prompting and finetuning

(05:53) Few-shot prompting

(06:48) Supervised finetuning

(10:54) Activation addition is not spiritually similar to finetuning

(12:24) Steering vectors dont seem to hurt capabilities much

(13:09) Progress in activation additions

[Linkpost] “Steering Llama-2 with contrastive activation additions” by Nina Rimsky, Wuschel Schulz, NickGabs, Meg, evhub, TurnTrout

LessWrong (30+ Karma)