Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Tutorial Session: Review of Q-Learning

In this lecture, Anikait, a TA for CS224R, provides an overview of Q-Learning, beginning with a brief review of Markov Decision Processes (MDPs) and transitioning to tabular problems using fitted queue iteration. The lecture progresses to Parametric Q-Learning, covering bias-variance trade-offs and comparing TD regression with Monte Carlo rollouts. Practical aspects of learning effective Q-functions are discussed, including replay buffers, overestimation issues, and dealing with the gradient effects of TD learning. Anikait uses a grid world problem to illustrate key concepts such as value functions, Q-functions, and advantage, and explains how to learn a Q-function through dynamic programming. The discussion also covers the differences between Monte Carlo and TD estimates, n-step returns, and techniques for stabilizing Q-Learning, such as semi-gradients, target networks, gradient clipping, Huber loss, and replay buffers to improve the stability and performance of Q-Learning algorithms.