📝Tweet, 📄 Paper

We introduce TasteVal, a benchmark that measures the experimental research taste of AI models on AI R&D tasks. We define research taste as the ability to pick interesting problems to solve, design experiments, and interpret experimental results. TasteVal measures the experimental component of research taste; given a fixed research problem, we measure how well a model iteratively designs experiments and draws conclusions from their outcomes. In the AI Futures Model, experimental compute becomes the main bottleneck once coding is automated, so the time to superintelligence depends primarily on how fast research taste improves.

Setup. Given a fixed AI R&D problem, TasteVal measures how well a model designs experiments and draws conclusions from their results. We operationalize experimental research taste as compute efficiency: how much experimental compute a model needs to reach a given score, relative to human experts. A model that reaches the experts' score with half the experimental compute has twice their experimental research taste. Under this definition, doubling a model's experimental research taste has the same effect as doubling the compute available to run its experiments.

TasteVal consists of 8 novel tasks representative of frontier AI R&D, spanning pretraining data curation, pretraining and fine-tuning [...]

---

First published:
October 6th, 2026

Source:
https://www.lesswrong.com/posts/bAGagjZAPgXEHmkAH/tasteval-measuring-the-experimental-research-taste-of-ai

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Scatter plot showing AI models' Experimental Research Taste increasing over release dates toward Human Expert level.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts” by Ollie J, danesherbs

LessWrong (30+ Karma)

“TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts” by Ollie J, danesherbs