Stanford CS224N NLP with Deep Learning | 2023 | Lecture 8 - Self-Attention and Transformers

Self-attention and transformer architectures represent a fundamental shift in modern natural language processing, addressing the inherent limitations of recurrent neural networks. RNNs struggle with long-distance dependencies and sequential processing, which hinders parallelization on GPUs. Self-attention resolves these issues by enabling direct interactions between all words in a sequence, regardless of their distance, while allowing for efficient parallel computation. Key components of the transformer architecture include multi-head attention, which captures diverse contextual information, and positional embeddings that restore sequence order. Further optimizations, such as residual connections and layer normalization, stabilize training and improve convergence. While transformers face quadratic computational complexity relative to sequence length, their ability to leverage massive datasets and parallel hardware has made them the standard for current AI systems.