Ep#107: Do As I Do: Dexterous Manipulation Data from Everyday Human Videos

"DO AS I DO" addresses the challenge of training robots for dexterous manipulation using large-scale internet video data. The proposed pipeline reconstructs 3D hand-object interactions from monocular videos, then retargets these motions onto robot hands within a physics simulator. This approach bypasses the need for massive, manually curated datasets by automating the extraction of physically consistent trajectories. Despite a low 4% acceptance rate for raw internet clips, the method successfully generates usable data for training policies. Key hurdles include handling occlusions, camera motion, and ensuring physical grounding in simulated environments. Future advancements rely on improving monocular depth estimation, leveraging generative video models for more controllable data, and bridging the gap between human-like hand kinematics and robot hardware constraints to achieve robust, generalized policies.