The is a slimmed down version which omits some extra examples but includes my theorizing about ChatGPT, my investigations of it, and my findings.

Epistemic status: Pretty uncertain. I think my conclusions are probably more right than wrong, more useful than harmful, and would especially benefit people with only an average or below-average understanding of LLMs. There may be prior art that I'm unaware of; if not, maybe this will provide a launching point for others to begin deeper investigations.

In Scott Alexander's latest link roundup, he wrote:

A surprising LLM failure mode: if you ask questions like “answer with a single word: were any mammoths still alive in December”, chatbots will often answer “yes”. It seems like they lack the natural human assumption that you meant last December, and are answering that there was some December during which a mammoth was alive. I find this weird because LLMs usually seem very good at navigating the many assumptions you need to communicate at all; this one stands as a strange exception.

I think I’ve got a satisfactory answer to this strange exception, but first I want to walk through my original theories, then go over the patterns I observed [...]