Robot Learning 2026 – Lecture 5: Reinforcement Learning II | ETH Zürich

Reinforcement learning focuses on parameterizing policies to maximize expected rewards directly, moving beyond value-based methods like Q-learning. The REINFORCE algorithm initiates this approach by using Monte Carlo sampling to estimate gradients, though it suffers from high variance and causality issues. These limitations are addressed by implementing "reward-to-go" to respect temporal causality and subtracting baselines to reduce variance. Advancing to advantage functions enables more precise policy updates. To improve sample efficiency, importance sampling allows for multiple gradient updates from a single data batch, while entropy regularization prevents premature convergence. Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) further stabilize training by constraining policy changes. Finally, Soft Actor-Critic (SAC) integrates these concepts into a fully off-policy actor-critic framework, utilizing replay buffers to achieve superior sample efficiency and robustness in complex robotic tasks like dexterous manipulation and locomotion.