Ep#50: EMMA: Scaling Mobile Manipulation via Egocentric Human Data

Human Egocentric Data Scales Mobile Robot Manipulation

Mobile manipulation imitation learning is bottlenecked by teleoperation, which is slow and hard. EMMA collects human demonstrations with Meta Aria glasses, treats navigation knowledge as directly transferable because the scene is shared, and co-trains mobile human data with static robot data. A phase splitter and kinematic retargeting make full-body human data usable, and the resulting policy beats a teleoperated Mobile ALOHA baseline while generalizing to new scenes.

Takeaways

• Human egocentric data replaces slow teleoperation. Using Meta Aria glasses, the authors collect human demonstrations in minutes and Meta's perception service returns hand pose, SLAM, point clouds and eye tracking. Teleoperating the same mobile task is far slower, taking about half a minute to pull a tissue from a box.
• Navigation transfers directly from humans to robots. Because the scene is the same, the robot's motion need only be a subset of the human's. Lawrence recorded himself walking between two tables and that transferred zero-shot, so only static robot data needs co-training with mobile human data.
• EMMA beats a teleoperated baseline and generalizes. It beats Mobile ALOHA trained on one hour of teleoperated data, 31 to 33 demonstrations, with the same HPT trunk. In a new scene, half an hour of human data is enough for grasping where the robot-only baseline misses the wine glass.
• Human data scales with a small robot mix. Handover wine success rises to about 30 percent as human data grows from 15 to 60 minutes, with 60 minutes equal to roughly 200 human demonstrations. The co-training mix is about 1 to 3 by dataset contribution and 1 to 1 per batch.
• Wearable glasses make human data collection fast. The lab has about 10 pairs of Aria glasses, and Michael Cho estimates that with 10 times as many glasses as robots, human data collection is about 30 times as fast. As smart glasses become popular, this wearable data could scale robot learning.
• Eye gaze is an untapped robotics modality. Lawrence says he does not know anyone in robotics using gaze, though it could help affordance prediction. Pranav sees it as a way to read intent, like looking left and right before crossing a street.

Engineering the human-to-robot gap

1. Phase splitter. A first three-layer MLP trained on five minutes each of manipulation and navigation was replaced with a hand-velocity-ratio heuristic and a Gaussian mixture model. It gives a near 100 percent accurate split with the same hyperparameters across four tasks.
2. Kinematic retargeting. Human trajectories are mapped to a virtual differential drive robot by optimizing command velocities with weights on position and yaw, plus a smoothness term. Without retargeting, the robot misses the human recipient in the handover task.