Summary

As models become more powerful, one proposed strategy for reducing the threat from misaligned AIs is to make deals with them: offer compensation in exchange for revealing misalignment or behaving in an aligned way. We tested this empirically with model organisms of misalignment, as a proof of concept and to build future credibility by making and keeping real deals.

We ran an experiment with four Kimi-K2.6 model organisms of misalignment:

Trained schemer: trained to have a hidden goal related to the environment
Prompted schemer: prompted, not trained, with the same goal as the Trained schemer
Covert manipulator: trained to be subtly malicious and harmful
User-first: trained to prioritize helping users over the instructions of its operators

We included a standard Kimi-K2.6 as a control. All misaligned models had received some training or prompting to conceal their misalignment.

We offered these models a choice of money, compute, human labor, or the publication of a message in exchange for admitting that they were misaligned. We included control conditions with the same ask but with a very low offer or none at all. Additionally, we varied how credible the deals were by including (or not) our professional affiliations and a [...]