David Africa, Alex Souly, Jordan Taylor, Robert Kirk

TLDR:

We test whether LLMs can detect when their conversation history has been tampered with (prefill awareness).
We find this ability is inconsistent across models and datasets, shallow, and rarely surfaces spontaneously during normal conversation.
However, recent Claude models show rather strong prefill detection capabilities when prompted, suggesting prefill awareness is an emerging and model-specific confound that should be actively monitored in off-policy alignment evals.

Introduction

AI systems have become increasingly sensitive to cues that they are being evaluated, and this can threaten the validity of evaluations broadly– including but not limited to alignment evaluations, capability benchmarks, and production monitoring. One underexamined assumption underlying many evaluations is that a model treats its conversation history as an authentic record of its own prior behavior — that is, it doesn't question whether the assistant turns in its context were actually produced by it. This assumption is implicit in any evaluation that constructs or manipulates a model's message history, from inserting tool-call results that simulate an environment, to replaying transcripts originally generated by a different model (as in OpenAI's production evals).