Ranjay Krishna on Why Robots Still Fail in the Real World and the Data Problem Holding Them Back

Scaling diverse, synthetic simulation environments provides a viable pathway for training general-purpose robots, effectively bridging the "sim-to-real" gap that has long hindered robotic deployment. Ranjay Krishna, an assistant professor at the University of Washington and researcher at the Allen Institute, argues that the difficulty in deploying robots stems from insufficient training data diversity rather than inherent simulation limitations. By generating hundreds of thousands of varied, annotated 3D environments, researchers can train models to handle complex, unstructured tasks like navigation and manipulation without relying on expensive, real-world data collection. This approach emphasizes "visual grounding," where robots map sensor inputs directly to motor actions, bypassing the need for large language model intermediaries. This methodology prioritizes long-term research goals, aiming to create open-source, foundational models that enable robots to adapt to unpredictable, real-world scenarios through learned motor intentionality.