Stanford CS229 Machine Learning | Spring 2026 | Lecture 14: Transformers, In-Context Learning

Large language models function as autoregressive systems that generate tokens sequentially based on preceding inputs. Tokenization serves as the foundational step, utilizing subword-level encoding to efficiently represent text as numerical IDs. These models decompose complex language distributions into products of conditional probabilities, parameterized by transformer architectures. The attention mechanism facilitates token interaction by calculating inner products between query and key vectors, which are masked to preserve the autoregressive constraint. While this architecture enables powerful contextual understanding, the $O(T^2)$ computational complexity of standard attention creates significant memory and processing challenges for long sequences. Consequently, architectural optimizations like flash attention are essential for managing computational overhead and memory footprints in production environments.