Stanford CS229 Machine Learning | Spring 2026 | Lecture 20: GMM (EM), PCA

Reinforcement learning algorithms like Policy Gradient and Proximal Policy Optimization (PPO) provide the mathematical foundation for training large language models to improve reasoning capabilities. Policy Gradient maximizes return by weighting trajectories based on rewards, while PPO stabilizes this process through importance sampling and clipping, preventing unstable, overly aggressive parameter updates. Integrating chain-of-thought reasoning allows models to generate intermediate thinking tokens, significantly enhancing performance on complex mathematical and logical tasks. Training these models involves treating generation as a Markov Decision Process, where rewards are assigned based on final answer accuracy. This approach often utilizes multiple sampled trajectories to establish a baseline, effectively bootstrapping the model's ability to produce logical, step-by-step reasoning processes without requiring exhaustive, manually labeled datasets.