My credence: 33% confidence in the claim that the growth in the number of GPUs used for training SOTA AI will slow down significantly directly after GPT-5. It is not higher because of (1) decentralized training is possible, and (2) GPT-5 may be able to increase hardware efficiency significantly, (3) GPT-5 may be smaller than assumed in this post, (4) race dynamics.

TLDR: Because of a bottleneck in energy access to data centers and the need to build OOM larger data centers.

The reasoning behind the claim:

Current large data centers consume around 100 MW of power, while a single nuclear power plant generates 1GW. The largest seems to consume 150 MW.
An A100 GPU uses 250W, and around 1kW with overheard. B200 GPUs, uses ~1kW without overhead. Thus a 1MW data center can support maximum 1k to 2k GPUs.
GPT-4 used something like 15k to 25k GPUs [...]

---

Outline:

(00:42) The reasoning behind the claim:

(03:18) Unrelated to the claim:

(04:23) How big is that effect going to be?

(06:52) Impact of GPT-5

---

First published:
April 26th, 2024

Source:
https://www.lesswrong.com/posts/xyL5kb8RBGLiupGLf/scaling-of-ai-training-runs-will-slow-down-after-gpt-5

---

Narrated by TYPE III AUDIO.

“Scaling of AI training runs will slow down after GPT-5” by Maxime Riché

LessWrong (30+ Karma)

“Scaling of AI training runs will slow down after GPT-5” by Maxime Riché