Measuring AI Ability to Complete Long Tasks - METR

In their original 2025 paper, METR noticed that the slope (aka task horizon doubling time) of the trendline for models released in 2024 and later is different from the slope for <2024 models.

First, I decided to check whether a piecewise linear function fits the data better than a simple linear function. If it doesn't, then this change in trend is a random fluke and there is nothing worth talking about.

Here is the data so far (SOTA models only):

The piecewise linear function clearly provides a better fit, based on quantitative metrics such as root mean square error, mean absolute error, R^2 and most importantly Bayesian information criteria (BIC); and based on a qualitative "bro just look at it" assessment. I added RMSE, MAE and R² as extra information, keep in mind that since a piecewise linear function has more parameters, it's not surprising that it fits data better than linear, this is to be expected. BIC penalizes model complexity (number of degrees of freedom), so it's more relevant for this case.

However, I'm not satisfied. Let's randomly remove 20% of datapoints from this graph, do it a few [...]

---

First published:
March 3rd, 2026

Source:
https://www.lesswrong.com/posts/9ihaaXqdc3i328tLp/i-m-confused-by-the-change-in-the-metr-trend

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Graph showing task time trends for AI models with exponential trendlines, 2019-2025.

Monte Carlo results showing piecewise better in 98% of 10,000 iterations.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“I’m confused by the change in the METR trend” by Expertium

LessWrong (30+ Karma)