Chelsea Finn: This is the State of the Art in Robotics

Physical Intelligence bets on one robot model for reliability

Physical Intelligence is betting that one generalist robot model, trained on all its data and refined with human interventions, can make physical AI reliable enough for real-world work. A robot acts directly in the world, so its mistakes are costly and reliability must be far higher than for recommendation or ad ranking systems. The company's Pi0.7 matches fine-tuned specialists, and its memory system extends autonomy to tasks that run for ten to fifteen minutes.

Takeaways

• Robots must be far more reliable. Unlike recommendation or ad ranking systems, where the customer can spot a mistake and ignore it, a robot acts directly in the world. The espresso task needed over 90% reliability to run without babysitting.
• Human interventions teach robots to recover. A person teleoperates the robot out of dead ends, like two boxes grabbed flush together. A general-purpose value function trained on robot video amortises the value estimate across prompts, rather than rolling out dozens of attempts per prompt. The RL stage gave about 2x throughput.
• One generalist model beats fine-tuned specialists. Pi0.7 trains on all Physical Intelligence data, including low-quality demonstrations, policy rollouts, human videos and web data, prompted with memory, instructions and optional subgoal images. It matches or outperforms fine-tuned Pi0.6 specialists and handled an air fryer seen in only three dataset episodes.
• Memory unlocks long-horizon robot autonomy. Most robot foundation models have no context. Passing ten seconds of video at 50 hertz from four cameras is half a million tokens, so Physical Intelligence uses short-term video plus text summaries of the past ten to fifteen minutes, enabling ten-to-fifteen-minute kitchen cleaning.
• Long autonomous runs are now achievable. The latte-making policy ran for thirteen hours straight. The robot folded clothes it had never seen in a home it had never seen, and on a bimanual UR5E platform with no folding data it approached human teleoperation.
• Robotics trails generalist AI by three years. Generalist AI moved from ImageNet in 2012 to pre-trained models, then GPT-2 and DALL-E. In 2023 robotics projects still collected bespoke datasets, and Physical Intelligence's answer is a single model trained across all its data and many embodiments.

How to break into robotics

• Fine-tune open-source generalist policies such as Pi0 and Pi0.5, which are already widely used, and reserve bespoke approaches for constrained settings like surgical robots in a basement operating room.
• There is a lot of software engineering in robotics, including the software stack that must run reliably on the robot, hardware, machine learning infrastructure and data infrastructure. A PhD is not needed for much of that work.