Ep#105: Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

"Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning" addresses the performance ceiling in behavior cloning by introducing a recursive self-improvement loop. Authors Varun Giridhar and Animesh Garg explain how a learned Q-function acts as a critic, ranking action chunks generated by a frozen behavior cloning policy during online inference. This method enables robots to learn from their own failures without requiring additional teleoperation data, effectively steering policies toward mastery in fine-grained tasks like inserting credit cards or stacking cups. By averaging the top-performing action chunks, the framework achieves significant success rate improvements—moving from 25% to 80% in real-world scenarios—while maintaining computational efficiency. This model-free approach prioritizes inference-time compute, allowing for autonomous policy refinement and reducing the reliance on human-in-the-loop data collection, ultimately providing a scalable path for improving complex robotic manipulation.