Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 4 - LLM Training

The lecture focuses on the training of Large Language Models (LLMs), contrasting traditional task-specific training with transfer learning. It covers pre-training LLMs on vast datasets to predict the next token, using datasets like Common Crawl, and introduces metrics like FLOPs to measure compute. The discussion highlights the importance of scaling model and training set sizes, referencing the Chinchilla law for optimal compute allocation. Addressing the challenges of pre-training, such as high costs and knowledge cutoff dates, the lecture transitions to strategies for efficient training, including data and model parallelism, Zero Redundancy Optimization (ZERO), and FlashAttention. It also explores quantization techniques like mixed precision training to reduce memory usage and accelerate computation. Finally, the lecture discusses fine-tuning, instruction tuning, and evaluation methods, including benchmarks and user preference rankings, as well as LoRa and quantized LoRa.