Interviewing Tri Dao and Michael Poli of Together AI on the future of LLM architectures

Transformers dominate AI due to their hardware efficiency and scalability, yet their quadratic attention cost limits long-sequence processing. Alternative architectures like state space models and modern recurrent neural networks offer potential breakthroughs by optimizing memory usage and computational complexity. Tri Dao and Michael Poli, researchers at Together AI, highlight how models such as Mamba and Striped Hyena leverage linear recurrence and specialized CUDA kernels to achieve performance competitive with Transformers. While Transformers remain the industry standard for general tasks, hybrid architectures that compose different layers show promise for specialized applications like genomics and long-context processing. Future progress hinges on data quality and the development of efficient hardware-aware primitives, suggesting a shift toward more complex, specialized model designs that extend beyond standard attention mechanisms.