He Co-Invented the Transformer. Now: Continuous Thought Machines [Llion Jones / Luke Darlow]

The conversation centers on the current state of AI research, particularly the perceived over-reliance on Transformer models and the need for more exploratory approaches. Llion Jones, involved in inventing the Transformer, expresses concern that the field is stuck in a local minimum, echoing a similar situation with recurrent neural networks before the Transformer breakthrough. He advocates for protecting research freedom, referencing Kenneth Stanley's ideas, and shares his company Sakana AI's philosophy. Luke Darlow introduces the Continuous Thought Machine (CTM), a novel recurrent model inspired by biological systems, emphasizing its native adaptive compute and potential for solving problems in more human-like ways. The discussion covers the CTM's architecture, including internal thought dimensions, neuron-level models, and synchronization mechanisms, relating these to reasoning, memory, and challenges like the ARC challenge.