This replication was done as part of the Second Look Fellowship by Arav Dhoot and supervised by Yixiong Hao and Zephaniah Roe. I am grateful to Andy Wang for their feedback. My code can be found here.

"So the answer should be A - at active promoters and enhancers."

"Let me reconsider the biology to justify D."

~ Claude Opus 4.8

TL;DR

This work replicates and extends Emmons et al.'s finding that CoT unfaithfulness mostly occurs on easy tasks. Across 11 models from 6 families (not just Gemini), models follow simple hints unfaithfully well above baseline, but complex hints that require computation are followed near baseline. This corroborates Emmons et al.'s findings. Key extensions:

Follow rate ≠ concealment. Monitorability risk is decomposable into cue-susceptibility and concealment among followers, and the two don't correlate.
Decode-necessity is model- and task-specific. It is not a property of task difficulty alone, so a CoT-monitoring safety case is per-model, not universal.
LLMs verbalize even when they don't have to. However, this appears to be a chosen behavior (likely from post-training), so it could vanish under optimization pressure against monitors.