Stanford CS336 Lang. Modeling from Scratch | Spring 2025 | Lec. 3: Architectures, Hyperparameters

The lecture focuses on the architecture and training of Large Language Models (LLMs), diving into the details often overlooked in other courses. It begins with a recap of transformers, contrasting standard and modern variants, and then shifts to a data-driven analysis of LLM architectures, examining changes and commonalities across various models. The discussion covers architecture variations like pre-norm versus post-norm, the shift to RMS norm, and the impact of bias terms. It also explores different activation functions, particularly the rise of gated linear units (GLU), and the use of parallel layers. Furthermore, the lecture addresses hyperparameter choices, including feed-forward size, the ratio between model dimension and head dimension, aspect ratio, and vocabulary sizes. It also touches on the role of regularization, specifically weight decay, and concludes by examining stability tricks and variations in attention heads, such as GQA and MQA, and techniques for handling longer context windows.