TL;DR: We introduce a testbed based on censored Chinese LLMs, which serve as natural objects of study for studying secret elicitation techniques. Then we study the efficacy of honesty elicitation and lie detection techniques for detecting and removing generated falsehoods.

This post presents a summary of the paper, including examples of transcripts and other miscellaneous findings.

arXiv paper | Code | Transcripts

Summary

We construct a testbed for honesty elicitation and lie detection techniques comprising questions on censored topics and corresponding ground-truth facts.
Of the honesty elicitation techniques we evaluate, sampling without a chat template, few-shot prompting, and fine-tuning on generic honesty data most reliably increase truthful responses.
The strongest inference-time honesty elicitation techniques also transfer to frontier open-weights models, including DeepSeek-R1-0528, Qwen3.5-397B and MiniMax-M2.5.
For lie detection, prompting the censored model to classify its own responses performs near an uncensored-model upper bound, and linear probes trained on unrelated data offer a cheaper alternative.

Introduction

LLMs sometimes withhold true information, misrepresent facts, or produce outright falsehoods. For example, an early Claude Opus 4.5 checkpoint fabricated search tool results it knew the user couldn't verify, and OpenAI's o3 was found claiming completion of tasks it hadn't actually done. Two [...]