Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 6: Q-Learning

The podcast discusses value-based reinforcement learning (RL) methods, focusing on Q-learning and its practical implementation. It begins with a recap of value and Q-functions, then introduces a thought exercise to explore policy formulation using Q-functions. The lecture covers iterative algorithms, off-policy action critics, and the Bellman Optimality Equation. It also addresses challenges like sparse rewards and continuous action spaces. The discussion extends to exploration policies such as Epsilon-Greedy and Boltzmann Exploration, and a full Q-learning algorithm is presented. Practical tips for stabilizing Q-learning, including using target networks (DQN), Double DQN to reduce overestimation, and N-step returns, are also covered. The podcast concludes with advice on when to use different online RL algorithms like CPO, SAT, and DQN, and highlights successful applications of Q-learning in Atari games and robot grasping systems.