Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.This is the real reward output for an OS preference model. The bottom "jailbreak" completion was manually created by looking at reward-relevant SAE features.

Preference Models (PMs) are trained to imitate human preferences and are used when training with RLHF (reinforcement learning from human feedback); however, we don't know what features the PM is using when outputting reward. For example, maybe curse words make the reward go down and wedding-related words make it go up. It would be good to verify that the features we wanted to instill in the PM (e.g. helpfulness, harmlessness, honesty) are actually rewarded and those we don't (e.g. deception, sycophancey) aren't.

Sparse Autoencoders (SAEs) have been used to decompose intermediate layers in models into interpretable feature. Here we train SAEs on a 7B parameter PM, and find the features that are most [...]

---

Outline:

(01:30) What are PMs?

(03:27) Finding High Reward-affecting Features w/ SAEs

(04:01) Negative Features

(04:19) I dont know

(05:47) Repeating Text

(07:35) URLs

(08:15) Positive Features

(08:33) (Thank you) No problem!

(10:02) Youre right. Im wrong.

(10:49) Putting it all together

(10:57) General Takeaways

(12:13) Limitations and Alternatives

(12:17) Model steering

(12:56) Limited Dataset

(13:17) Later Layer SAEs Sucked!

(13:32) Small Token-Length Datapoints

(13:47) Future Work

(16:31) Technical Details

(16:35) Dataset filtering

(17:10) Attribution Patching

(18:20) SAEs

---

First published:
July 1st, 2024

Source:
https://www.lesswrong.com/posts/5XmxmszdjzBQzqpmz/interpreting-preference-models-w-sparse-autoencoders

---

Narrated by TYPE III AUDIO.

“Interpreting Preference Models w/ Sparse Autoencoders” by Logan Riggs, Jannik Brinkmann

LessWrong (30+ Karma)