Robot Learning 2026 – Lecture 7: Sequence Modeling & Transformers | ETH Zürich

Sequence modeling and transformer architectures provide a robust framework for addressing the limitations of reactive robotic policies, which struggle with memory constraints and action jitter. By treating robotics as a multi-modal sequence prediction problem, models integrate language, vision, and action tokens to reason over entire trajectories rather than isolated snapshots. Key advancements include using Discrete Cosine Transform (DCT) and Byte-Pair Encoding (BPE) to compress continuous action data, enabling high-frequency control without the performance degradation seen in naive autoregressive approaches. Scaling these models—informed by principles like the Chinchilla optimal compute frontier—allows for the emergence of dexterous, long-horizon manipulation capabilities. Integrating visual and linguistic modalities through early fusion, late fusion, or native multimodal training further enhances a robot's ability to perceive and interact with complex, unseen environments, effectively applying the "bitter lesson" of scaling compute to robotic control.