This is a linkpost for https://1drv.ms/b/s!AkbNRQZiRerJ60FS9KJ1qqIo9qyC?e=2AqBeH

(The link above is likely going to expire in favor of arxiv once I get it submitted. Incidentally, if anyone wants to endorse me for cs.LG on arxiv, please DM me, and thanks!)

Abstract:

To help evaluate and understand the latent capabilities of language models, this paper introduces an approach using optimized input embeddings, or `soft prompts,' as a metric of conditional distance between a model and a target behavior. The technique aims to facilitate latent capability discovery as a part of automated red teaming/evaluation suites and to provide quantitative feedback about the accessibility of potentially concerning behaviors in a way that may scale to powerful future models, including those which may otherwise be capable of deceptive alignment. An evaluation framework using soft prompts is demonstrated in natural language, chess, and pathfinding, and the technique is extended with generalized conditional soft prompts to aid [...]

---

Outline:

(00:22) Abstract:

(01:13) What is this?

(01:57) Why is this useful?

(03:06) Are there any cute short results for me to look at?

(04:19) Why did you make this?

(06:13) Whats next?

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
February 2nd, 2024

Source:
https://www.lesswrong.com/posts/yCZexC2q2XEeWWiZk/soft-prompts-for-evaluation-measuring-conditional-distance

Linkpost URL:
https://1drv.ms/b/s!AkbNRQZiRerJ60FS9KJ1qqIo9qyC?e=2AqBeH

---

Narrated by TYPE III AUDIO.

[Linkpost] “Soft Prompts for Evaluation: Measuring Conditional Distance of Capabilities — LessWrong” by porby

LessWrong (30+ Karma)