This post is prompted by two recent pieces:

First, in the podcast "Emergency Pod: Mamba, Memory, and the SSM Moment", Nathan Labenz described how he sees that we are entering the era of heterogeneity in AI architectures because currently we have not just one fundamental block that works very well (the Transformer block), but two kinds of blocks: the Selective SSM (Mamba) block has joined the party. These are natural opposites on the tradeoff scale between episodic cognitive capacity (Transformer's strong side) and long-term memorisation (selective SSM's strong side). So, we will probably quickly see the emergence of complicated hybrids between these two, trying to get the best from both types of blocks[1].

This reminds me of John Doyle's architecture theory that predicts that AI architectures will evolve towards modularisation and component heterogeneity, where the properties of different components (i.e., their positions at different tradeoff spectrums) will converge to [...]

---

Outline:

(04:36) Implications for AI Safety RandD

(05:42) System-level explanation and control frameworks, mechanism design

(07:01) Implications for AI policy and governance

The original text contained 6 footnotes which were omitted from this narration.

---

First published:
December 27th, 2023

Source:
https://www.lesswrong.com/posts/Btom6dX5swTuteKce/agi-will-be-made-of-heterogeneous-components-transformer-and

---

Narrated by TYPE III AUDIO.

“AGI will be made of heterogeneous components, Transformer and Selective SSM blocks will be among them” by Roman Leventov

LessWrong (30+ Karma)