Ep#51: Humanoid Everyday

A 10,300-trajectory humanoid dataset shows where VLAs fail

Humanoid manipulation lacks data, so a USC team spent nine months collecting 3m frames on Unitree G1 and H1 robots and released it as Humanoid Everyday. Benchmarks built on it show pretrained VLA models beat imitation learning, and that a prior on this data lifts pi 0.5 from zero to four successes on precision insertion. But insertion still defeats nearly every policy, and the teleoperation used to collect the data does not scale.

Takeaways

• Humanoid Everyday spans 260 tasks and 10,300 trajectories. Collected on the Unitree G1 and H1, the open-world dataset holds over 3m frames across seven categories, from basic and deformable manipulation to tool use, articulated manipulation, high-precision work, human-robot interaction and locomotion.
• Pretrained VLA models beat imitation learning baselines. GR00T N1.5 was the strongest policy, and pretrained VLA models beat the traditional imitation learning baselines on average. High-precision insertion defeated nearly every policy, and only OpenVLA reached about 10% success.
• Pretraining on the dataset lifts hard-task success. The team ran an ablation on insertion. pi 0.5 scored no successes without the Humanoid Everyday prior and four out of ten with it, evidence that the data transfers to tasks the policies otherwise fail.
• Teleoperation cannot scale humanoid data collection. Yue Wang says whole-body teleoperation is still unsolved. The Vision Pro rig could not control the pelvis and legs together, so Unitree's off-the-shelf policy handled the lower body, and she points to UMI-style data, egocentric human video, simulation and generative models instead.
• Humanoid action spaces break existing tooling. A humanoid has 28 action dimensions against about seven for a robot arm. The G1 hand's tactile sensor adds 66 dimensions, hard to pair with that space, and the team plans an action tokenizer that compresses a 6 by 28 horizon into about 20 tokens.
• Nine months of work and two broken G1s. The project began in early January and ran almost nine months. The usable data is about 30 hours, and 20 to 30 percent was abandoned early because of wrong timestamps and frames. Operators needed under two hours of training.

How the data was collected

1. The team rewrote Unitree's data collection pipeline, which was too slow for large data streams.
2. Teleoperators wore an Apple Vision Pro, and inverse kinematics turned the tracked hand motion into joint poses for the humanoid.
3. Depth, LiDAR, RGB, tactile and proprioception streams were synchronized at 30 hertz through Python shared memory, with a separate LiDAR server.