Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 15: Hierarchical RL and IL

The podcast discusses the use of hierarchy in training AI systems for long-horizon tasks, where tasks are broken down into subtasks for easier management. It covers key design choices in hierarchical imitation and reinforcement learning, including representing skills and goals, determining when to transition between tasks, and supervising different levels of the hierarchy. The discussion includes examples such as cooking, autonomous driving, and language models, and it also explores the benefits of hierarchical approaches over flat policies, including improved knowledge sharing, structured exploration, and computational efficiency. The podcast further examines design choices like goal representation, supervision levels, and task transition strategies, and it presents example systems using language and images as intermediate representations, highlighting the adaptation of high-level and low-level policies to account for deficiencies.