Radical Talks, Masterclass Edition: Jim Fan on The Future of Robotics and Embodied AI

NVIDIA Bets Simulation Can Give Robots Their Internet

Robotics is stuck where language models were before the internet, with no web of experience to pretrain on. NVIDIA's answer, per Jim Fan, is to manufacture that experience in simulation and in world models that predict pixels, then compress it into tiny policies that run in milliseconds. His Minecraft work showed agents can accumulate skills in context without fine-tuning, but low-level motor control remains the hard part. He expects structured factories first and homes last.

Takeaways

• Voyager showed agents can learn without fine-tuning. GPT-4 received a Minecraft API to read instructions, write code, debug itself and store working code in a repository. Fan calls this in-context lifelong learning, and Minecraft's Turing-complete blocks supply effectively infinite training diversity.
• Low-level control is robotics' hardest unsolved problem. Voyager sidestepped perception and motor control by turning Minecraft into a code API. Fan says system two reasoning improves with more data and environments, while system one, fast reactive control, is harder and its data far harder to collect.
• Robot data is scarce and teleoperation cannot scale. At NVIDIA headquarters a researcher in VR mirrors her hands onto a robot, capped at 24 hours a day in theory and about 4 hours of good data in practice. Roboticists have no internet to pretrain on.
• Simulation and randomization carry the robot data load. Eureka teaches robot hands to spin pens purely through reinforcement learning in GPU simulation running 10,000 times faster than real time, and domain randomization across 10,000 environments lets policies transfer. Humanoids absorb ten years of training in two hours.
• World models generate training data from pixels. World models are pretrained video foundation models that take actions and predict future pixels, so scene complexity costs them nothing. A robot video shown was generated entirely by GR00T Dreams. NVIDIA open sourced GR00T N1, a vision-language system two with a diffusion system one above 100 hertz.
• The physical Turing test is the next challenge. The milestone Fan proposes is a wrecked house cleaned and a candlelit dinner prepared, with no way to tell whether a human or a robot did it. Moravec's paradox explains why the tasks easy for humans are incredibly hard for machines.

Where robots land first

1. Factories come first. Industrial settings are regular and structured, and robots can be walled off from people, so safety demands are lower.
2. Wet labs and agentic fleets follow. A central reasoning model breaks a task into pieces and distributes it across a fleet, for assembly lines and multistage experiments.