18:I[66739,["/_next/static/chunks/0u0nyy7wb9me2.js?dpl=dpl_EVxQmm5ee6SfSwpTLEi7dkcqj48i","/_next/static/chunks/05hx6kuf6-oy4.js?dpl=dpl_EVxQmm5ee6SfSwpTLEi7dkcqj48i","/_next/static/chunks/12hmyxte1w1dw.js?dpl=dpl_EVxQmm5ee6SfSwpTLEi7dkcqj48i","/_next/static/chunks/166.u_60qj09..js?dpl=dpl_EVxQmm5ee6SfSwpTLEi7dkcqj48i","/_next/static/chunks/0g9_wl34k-6l4.js?dpl=dpl_EVxQmm5ee6SfSwpTLEi7dkcqj48i","/_next/static/chunks/06dn~uer6.43l.js?dpl=dpl_EVxQmm5ee6SfSwpTLEi7dkcqj48i"],"default"] 1a:I[38632,["/_next/static/chunks/0qqtp3r0h7kah.js?dpl=dpl_EVxQmm5ee6SfSwpTLEi7dkcqj48i","/_next/static/chunks/0iukvycbd42rl.js?dpl=dpl_EVxQmm5ee6SfSwpTLEi7dkcqj48i"],"IconMark"] 19:T4e9,

The provided text details the development of Titans, a new family of neural architectures designed to overcome the fixed context window and computational scaling issues inherent in modern Transformers. The core of this system is a novel neural long-term memory module that operates as a meta in-context learner, dynamically updating its weights based on a calculated "surprise" metric that includes both momentum and an adaptive forgetting mechanism. This mechanism allows the module to store persistent, historical information, thereby acting as long-term memory while the attention component handles immediate context as short-term memory. The paper presents three distinct architectural ways to integrate this new component: Memory as a Context (MAC), Memory as a Gate (MAG), and Memory as a Layer (MAL). Empirical testing across diverse tasks, including language modeling and extreme-length reasoning benchmarks, demonstrates that Titans achieve superior performance and efficiency compared to baseline models, successfully scaling to context window sizes exceeding two million tokens.

Titans: Long-Term Neural Memory for Scaling Context

Steven AI Talk

Titans: Long-Term Neural Memory for Scaling Context