FutureHouse is a company that builds literature research agents. They tested it on the bio + chem subset of HLE questions, then noticed errors in them.

The post's first paragraph:

Humanity's Last Exam has become the most prominent eval representing PhD-level research. We found the questions puzzling and investigated with a team of experts in biology and chemistry to evaluate the answer-reasoning pairs in Humanity's Last Exam. We found that 29 ± 3.7% (95% CI) of the text-only chemistry and biology questions had answers with directly conflicting evidence in peer reviewed literature. We believe this arose from the incentive used to build the benchmark. Based on human experts and our own research tools, we have created an HLE Bio/Chem Gold, a subset of AI and human validated questions.

About the initial review process for HLE questions:

[...] Reviewers were given explicit instructions: “Questions should ask for something precise [...]

---

First published:
July 29th, 2025

Source:
https://www.lesswrong.com/posts/JANqfGrMyBgcKtGgK/about-30-of-humanity-s-last-exam-chemistry-biology-answers

---

Narrated by TYPE III AUDIO.

“About 30% of Humanity’s Last Exam chemistry/biology answers are likely wrong” by bohaska

LessWrong (30+ Karma)

“About 30% of Humanity’s Last Exam chemistry/biology answers are likely wrong” by bohaska