"I Co-Invented the Transformer. Now I'm Replacing It." & Continuous Thought Machines - Llion Jones and Luke Darlow [Sakana AI]

The podcast features Llion Jones, one of the inventors of the Transformer model, and Luke Darlow, a research scientist at Sakana AI, discussing the current state and future directions of AI research. Jones expresses concern about the field being oversaturated with Transformer-based research, advocating for more exploratory work. Darlow introduces the Continuous Thought Machine (CTM), a new recurrent model with native adaptive compute, drawing inspiration from biological systems. They explore the limitations of current AI models, particularly their "jagged intelligence" and the tendency to brute-force solutions rather than developing genuine understanding. Jones and Darlow highlight the importance of research freedom and the potential for AI to drive future scientific progress, emphasizing the need for models that can reason more like humans. The conversation also covers the Sudoku Bench dataset as a challenging reasoning benchmark and the potential of the CTM architecture to address limitations in current language models.