tl;dr: LLMs rapidly improving at software engineering and math means lots of projects are better off as Google Docs until your AI agent intern can implement them.

Implementation keeps getting cheaper

Writing research code has gotten a lot faster over the past few years. Since 2021 and OpenAI Codex, new models and tools such as Cursor built around them have saved myself more and more time on coding every year.

This trend is accelerating fast: AI agents using Claude-3.5-Sonnet and o1-preview can do tasks that take ML researchers up to 2 hours of coding. This is without considering newer models such as o3, which do 70% on SWE-bench out of the box.
Yet this progress remains somewhat concentrated in implementation: progress on “soft” skills like idea generation has, as far as I can tell, been slower.

I’ve come to believe that, if you work in technical [...]

---

Outline:

(00:18) Implementation keeps getting cheaper

(01:33) When later means better

(03:01) Did I do well in 2024?

(07:29) Themes for temporally privileged work

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
January 7th, 2025

Source:
https://www.lesswrong.com/posts/fsLpvRiLt76pcCcPD/you-should-delay-engineering-heavy-research-in-light-of-r

---

Narrated by TYPE III AUDIO.

“You should delay engineering-heavy research in light of R&D automation” by Daniel Paleka

LessWrong (30+ Karma)