Wearable system captures human motion data to train humanoid robots

Researchers at the Shanghai AI Laboratory, Zhejiang University and Shanghai Jiao Tong University have developed a wearable capture system and a 100-hour dataset, called EgoExplorer, to record human loco-manipulation for training humanoid robots. The system combines a head-mounted GoPro HERO13 camera with five VIVE Ultimate Trackers, sensor-equipped MANUS Metagloves or Wuji gloves, and custom 3D-printed mounts and straps. It captures egocentric video, whole-body motion, fine-grained finger articulation, tactile observations and step-level task descriptions while people perform professional activities. Because existing datasets often provide only partial views of how locomotion, posture and object interaction are coordinated, the team built a pipeline named Calibrate-Complete-Constrain to reconstruct full-body movement from sparse sensor data. The pipeline first calibrates participant geometry and sensor placements, then completes untracked joint motion using learned movement patterns, and finally constrains the results to match sensor measurements, remain smooth and respect human body shapes. The researchers report that, against tracker-fitted references, the method reduced body-point error by 71.9% compared with the strongest baseline, with additional gains in hand stability and contact estimation. Preliminary experiments also showed that jointly generating video, motion and touch data outperformed generating each modality separately. In simulation, the team retargeted two human demonstrations to a Unitree G1 humanoid, producing controllers that followed the motion sequences without falling, although the study did not test physical robots or demonstrate precise object interaction on unfamiliar tasks. EgoExplorer could later support more advanced humanoid robot training by providing realistic multimodal examples of coordinated body, hand and tactile behavior in real-world environments.