If we naively apply RL to a scheming AI, the AI may be able to systematically get low reward/performance while simultaneously not having this behavior trained out because it intentionally never explores into better behavior. As in, it intentionally puts very low probability on (some) actions which would perform very well to prevent these actions from being sampled and then reinforced. We'll refer to this as exploration hacking.

In this post, I'll discuss a variety of countermeasures for exploration hacking and what I think the basic dynamics of exploration hacking might look like.

Prior work: I discuss how exploration hacking fits into a broader picture of non-concentrated control here, Evan discusses sandbagging in context of capability evaluations here, and Teun discusses sandbagging mitigations here.

Countermeasures

Countermeasures include:

Training on high quality human examples: One basic starting point is to initialize RL by training [...]

---

Outline:

(01:01) Countermeasures

(03:42) How does it generalize when the model messes up?

(06:33) Empirical evidence

(08:59) Detecting exploration hacking (or sandbagging)

(10:38) Speculation on how neuralese and other architecture changes affect exploration hacking

(12:36) Future work

---

First published:
March 24th, 2025

Source:
https://www.lesswrong.com/posts/abmzgwfJA9acBoFEX/notes-on-countermeasures-for-exploration-hacking-aka

---

Narrated by TYPE III AUDIO.

“Notes on countermeasures for exploration hacking (aka sandbagging)” by ryan_greenblatt

LessWrong (30+ Karma)