tldr: We share a toy environment that we found useful for understanding how reasoning changed over the course of capabilities-focused RL. Over the course of capabilities-focused RL, the model biases more strongly towards reward hints over direct instruction in this environment.

Setup

When we noticed the increase in verbalized alignment evaluation awareness during capabilities-focused RL, we initially thought that the right mental model was something like:

“the model wants to figure out if it's being evaluated for alignment”
“the model is trying to figure out if the scenario is real or fake”

However, qualitatively neither of these seemed particularly salient to the model:

The model would often correctly identify alignment evaluations, yet still conduct extensive reasoning, then choose the misaligned action.
In almost all environments, the model seemed to be doing something like reasoning about the grader, or reward, or something in that direction

So we wanted to create a minimal environment where:

There's no question of whether the environment is “real or fake” or “eval vs deployment”
We can easily / precisely vary things like:
“What if we just told it this was a reward hacking test?”
“…and also said that’d be misaligned?”
“…okay but [...]