Stanford CS336 Language Modeling from Scratch | Spring 2025 | Lecture 15: Alignment - SFT/RLHF

The lecture focuses on post-training techniques for large language models, transitioning from pre-training to making models useful and safe. It covers two main areas: supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). The discussion includes the importance of high-quality data, the challenges in collecting it, and the trade-offs between capabilities and safety. The lecture also touches on algorithmic questions, such as how to effectively use different types of data and scale up the training process, and it introduces methods like Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO). The content also explores the nuances of instruction tuning, including the potential for models to hallucinate and the impact of data length and style on model behavior.