Key points
- EgoZero learns robot policies zero-shot from egocentric human video on Aria glasses, folding towels at 60-70% success in 20 minutes.
- It maps the index-thumb midpoint to a Franka gripper and skips end-effector rotation, limited by a centimeter of hand detection error.
- Vincent Liu and Ademi Adeniji want to remove correspondence and fuse 3D with VLAs, and Adeniji stresses ergonomics and simplicity for scaling.
Robot manipulation policies learned from smart glasses video
Robot manipulation data does not scale through teleoperation or handheld rigs, so EgoZero bets on the sensors people already wear. Smart glasses capture egocentric video of anyone doing work, and a shared 3D point representation lets a policy trained on human hands drive a Franka arm. Twenty minutes of in-the-wild data already folds towels most of the time. The authors now pursue this as a company.
Takeaways
- Teleoperation and handheld demos do not scale. Teleoperation is high friction and needs trained operators, and Sim2Real breaks in the real world. UMI-style handheld demonstrations are unnatural and slow, while glasses are already worn, so anyone can collect data in their own environment.
- EgoZero learns policies from human glasses video. The contribution is showing that a unified 3D point space, borrowed from Point Policy, works zero-shot from egocentric in-the-wild data. Points in the egocentric frame let the policy predict 3D motion and let the camera sit anywhere at inference.
- Twenty minutes of data folds a towel. Across four settings, none in a lab, EgoZero reaches roughly 60-70% success folding a towel. It works while inference points fall inside the augmented training distribution and fails at the edge of that support volume, so augmentation drives generalisation.
- Hand tracking and correspondence cap accuracy. HaMeR plus Aria hand detection carries about a centimetre of error, and Adeniji says LiDAR or a stereo pair would give the ground-truth depth they want. The largest error source is the correspondence model, which matches Stable Diffusion activation features.
- Retargeting leaves the morphology gap partly open. The Franka has a two-finger gripper and a human has five, so EgoZero maps the index-thumb midpoint to the gripper midpoint and never learns end-effector rotation. A dexterous hand closer to human size would remove most of the gap.
- Ergonomics decides whether data collection works. A Manus glove and chest-mounted iPhone rig failed, and attempts to get depth from MASt3R-SLAM or structure from motion also fell short. Adeniji says complex setups had so many failure points that gathering enough data for a paper became the bottleneck.
- Real-world RL will not explore much. Adeniji wants autonomous, curiosity-driven real-world learning, but Liu notes that task-completion rewards are still zeros and ones, so a policy will not stray far from its starting behaviour. A decent imitation-learned policy comes first.
How EgoZero turns glasses video into a policy
- Label 2D object points on one expert image and track them through the demonstration video.