One-sentence Summary

NVIDIA researchers propose SONIC, a foundation model for humanoid motion tracking that scales network size (up to 42M parameters), data (100M+ frames from 700 hours of motion capture), and compute (21k GPU hours) to learn natural whole-body control from dense mocap supervision without manual reward engineering, thereby enabling real-time kinematic planning, VR teleoperation, and VLA-driven loco-manipulation.

Key Contributions

• The paper identifies motion tracking as a scalable foundational task for humanoid control, demonstrating favorable scaling with compute and data diversity. Scaling to 21,000 GPU hours and 100 million motion frames yields a controller that achieves universal tracking of diverse human behaviors.
• This work introduces a real-time kinematic motion generator for interactive control and a universal token space with specialized encoders for robot, human, and hybrid motion inputs, all mapped into a shared quantized representation.
• A comprehensive evaluation shows scaling trends, zero-shot transfer to unseen motions, robust sim-to-real deployment on physical humanoid robots, and successful integration with foundation models. The universal token space further enables VLA-driven whole-body loco-manipulation, including tasks requiring coordinated hand grasping and precise foot placement, across five real-world tasks.

Introduction

The authors tackle the longstanding challenge of scaling humanoid robot control, where state-of-the-art policies remain small MLPs trained on a handful of GPUs for single tasks and rely on hand-engineered rewards that do not transfer across behaviors. Although foundation models in language and vision have demonstrated that scaling unlocks emergent capabilities, humanoid control has been limited by the difficulty of designing generic objectives: locomotion rewards carry no signal for dancing or getting up, and prior generative imitation methods suffer from mode collapse as motion datasets grow in diversity. To overcome this, the authors identify physics-based motion tracking as a scalable foundational task, leveraging large-scale human motion capture data to provide dense supervision without reward engineering. They supersize training to 128 GPUs and 100 million motion frames, achieving universal tracking across diverse human behaviors. Beyond tracking, they introduce a real-time kinematic motion generator and a universal token space that unifies robot, human, and hybrid motion inputs, enabling the same controller to handle teleoperation, interactive character control, and integration with vision‑language‑action models for whole‑body loco‑manipulation.

Dataset

The authors construct a large-scale humanoid motion dataset from motion-capture recordings, then retarget and filter it for a Unitree G1 robot. Here is how the dataset is composed, processed, and used:

• Source data and filtering

  • Raw collection: approximately 700 hours of motion-capture clips (1–180 seconds) covering locomotion, daily activities, gestures, combat, and more.
  • Balanced male and female performers, thousands of unique behaviors, multiple subjects and takes per action.
  • Retargeted to the Unitree G1 using GMR and PyRoki; physically implausible motions (e.g., stair climbing, seated activities) are removed.
  • Final training corpus: 611 hours, over 100 million frames at 50 Hz.

• Dataset splits and diversity

  • 33 motion categories (see Table 2): basic/advanced locomotion, dance styles, gestures, combat, object manipulation, tool use, injured gait, stylistic variations (drunk, zombie, stealth), role-play, etc.
  • All motions are mirrored to produce left/right variants.
  • Training set: 8,447 unique motion sub-categories (611 hours).
  • Test-content split: entirely novel motion sub-categories not seen during training.
  • Test-repetition split: different takes and actor performances of known content, testing repetition generalization.

• Public subset: BONES-SEED

  • Released on Hugging Face: 142,220 annotated sequences (288 hours) from 522 actors.
  • Provided in SOMA and Unitree G1 formats, with natural language descriptions, temporal segmentation labels, and actor metadata.

• How the data is used

  • The full 611-hour training set is used for learning.
  • The two test splits evaluate generalization to unseen motion content and to new repetitions of familiar actions.
  • No additional cropping or mixture ratios are specified; the main processing steps are retargeting, plausibility filtering, and mirroring.

Method

The authors construct a large-scale motion dataset from motion-capture collections, spanning a broad spectrum of human behaviors including locomotion, daily activities, gesturing, and combat motions. The dataset provides rich intra- and inter-subject variation, as illustrated in the figure below.

After retargeting the source data to the Unitree G1 robot and filtering out physically implausible motions, the authors yield 611 hours of training data covering 33 motion categories. Explicit train and test splits are constructed to isolate novel motion content and novel repetitions of known content.

The core of the method is SONIC, a universal humanoid motion tracking framework. As shown in the figure below, it employs a unified control policy to track diverse motion commands from multiple input formats, seamlessly handling robot motion, human motion, and hybrid motion through a shared latent representation.

The humanoid motion tracking is formulated as a Markov Decision Process $\mathcal{M} = \langle \boldsymbol{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \boldsymbol{\gamma} \rangle$M=⟨S,A,T,R,γ⟩. The policy is trained using proximal policy optimization (PPO) to maximize the expected cumulative discounted return $\mathbb{E} \left[ \sum_{t=1}^{T} \gamma^{t-1} r_t \right]$E[∑t=1T​γt−1rt​].

The state representation $s_t$st​ comprises proprioceptive sensing $s_t^p$stp​ and a motion command $s_t^g$stg​. Proprioceptive information includes joint pose, joint velocity, root angular velocity, gravity vector, and previous action, concatenated over a 10-step history to provide temporal context. The motion command can be robot motion $g_r$gr​, human motion $g_h$gh​, or hybrid motion $g_m$gm​. The policy outputs target joint positions $a_t$at​, which are tracked by proportional-derivative controllers.

The reward function $r_t = \mathcal{R}(s_t^p, s_t^g) + \mathcal{P}(s_t^p, a_t)$rt​=R(stp​,stg​)+P(stp​,at​) combines tracking rewards and penalty terms. The tracking term minimizes errors in root position and orientation, body link positions and orientations, body link velocities, and end-effector positions. Penalties are applied for shaking and foot acceleration. To enhance robustness, systematic domain randomization is applied to physical parameters and motion commands during training.

The Universal Control Policy utilizes specialized encoders to process heterogeneous inputs into a shared latent space. Three multi-layer perceptron encoders handle robot motion $\mathcal{E}_r$Er​, human motion $\mathcal{E}_h$Eh​, and hybrid motion $\mathcal{E}_m$Em​. The encoded latent representation is quantized into a universal token $z$z using Finite Scalar Quantization (FSQ). FSQ is chosen to avoid codebook collapse and provide clean straight-through gradient estimation compatible with joint PPO optimization.

Two decoders process the universal token. The robot control decoder $\mathcal{D}_c$Dc​ takes the token and proprioceptive state to generate motor commands $a_t = \mathcal{D}_c(z, s_t^p)$at​=Dc​(z,stp​). An auxiliary robot motion decoder $\mathcal{D}_r$Dr​ reconstructs the robot motion command $\hat{g}_r = \mathcal{D}_r(z)$g^​r​=Dr​(z) to facilitate feature learning and serve as an implicit retargeting module.

Training optimizes four losses jointly in a single end-to-end loop:

$$\mathcal{L} = \mathcal{L}_{\text{ppo}} + \mathcal{L}_{\text{recon}} + \mathcal{L}_{\text{token}} + \mathcal{L}_{\text{cycle}}$$L=Lppo​+Lrecon​+Ltoken​+Lcycle​

The reconstruction loss $\mathcal{L}_{\text{recon}}$Lrecon​ ensures fidelity across input modalities, acting as a retargeting loss when the input is human motion. The token alignment loss $\mathcal{L}_{\text{token}}$Ltoken​ enforces pairwise alignment between all three encoder outputs. The cycle consistency loss $\mathcal{L}_{\text{cycle}}$Lcycle​ reinforces latent space coherence by ensuring translation from human to robot motion and back preserves essential characteristics. Asymmetric actor-critic training is employed, where the critic observes privileged simulation states while the actor operates on deployment-available observations.

For interactive control, the authors employ a generative kinematic motion planner trained as an autoregressive motion in-betweening model. Motion is represented using pelvis-relative joint positions and global joint rotations. Planning occurs in a latent space where continuous motions are encoded into latent tokens. The in-betweening process uses masked token prediction, where the neural backbone iteratively predicts and finalizes tokens with the highest confidence given context and target keyframes. A critically damped spring model generates root position and heading keyframes from user commands to improve behavioral predictability.

For multi-modal control involving video, text, and music, the system integrates GEM, a diffusion-based model that generates human motion sequences from mixed conditions. These sequences are fed into the SONIC framework via the human motion encoder. Deployment on the Unitree G1 platform utilizes a multi-rate architecture with concurrent loops for policy inference, command streaming, operator input, and kinematic planning, running on a Jetson Orin GPU.

Experiment

The Unitree G1 humanoid is used to evaluate SONIC's motion tracking, interactive control, multi-modal teleoperation, and loco-manipulation, showing that scaling data, model size, and compute yields strong generalization to unseen motions and high real-world success. A kinematic planner enables responsive whole-body control for navigation, boxing, and other skills, while a universal token space unifies video, text, music, and VR interfaces without retraining. When paired with a VLA model, the system performs complex tasks like coordinated hand and foot placement, demonstrating that scaled motion tracking provides versatile whole-body priors for autonomous humanoid control.

The GR00T N1.5 VLA model achieved a 75% average success rate across five whole-body loco-manipulation tasks when predicting universal motion tokens. Success varied widely, from 60% for soda can disposal to 95% for picking up a scrub, with the simpler 3-point interface task reaching 90%. The model successfully used its feet as manipulators, opening a trash can by stepping on a pedal with 70% success. Under the same multi-object policy and 3,900 training trajectories, object pickup success differed sharply: 95% for a scrub versus 75% for a carrot, indicating that object-specific factors strongly influence performance. The open trash can task, which demands precise foot placement and single-leg balance to press a pedal, reached 70% success, demonstrating the VLA's capacity to coordinate feet as manipulators.

The training set comprises 611 hours of motion across 8,447 unique sub-categories spanning 33 main categories. Test-content evaluates generalization to 182 sub-categories entirely absent from training, while test-repetition assesses robustness to new performances of known motion types, sharing all 1,088 sub-categories with training but no overlapping clips. Training data includes 8,447 sub-categories, far exceeding the 182 novel sub-categories in test-content and the 1,088 known sub-categories in test-repetition. Test-content has zero sub-category overlap with training, isolating generalization to unseen motion types. Test-repetition shares 100% of its sub-categories with training but uses completely disjoint clips, testing robustness to new takes and actor performances.

Using FSQ tokens as the action space yields substantially higher task completion success rates than explicit SMPL poses, with an average improvement of 42 percentage points. The benefit is most pronounced on complex, long-horizon tasks where SMPL poses fail completely. The compact, discrete token representation is easier for a VLA to learn from demonstrations, avoiding the error amplification seen with high-dimensional continuous poses. The success rate advantage of FSQ tokens over SMPL poses grows with task complexity, from +15 points on carrot pickup to +60 points on soda-can-to-trash-can. FSQ tokens provide a compact, discrete action space that avoids the large tracking failures caused by small prediction errors in the high-dimensional SMPL pose space.

FSQ consistently outperforms VQ-VAE in motion tracking, and increasing token dimension yields larger improvements than adding quantization levels. All three encoder types achieve success rates above 99.2%, and the human encoder's tracking error is only 0.6 mm higher than the robot encoder's, demonstrating effective cross-encoder alignment. FSQ reduces MPJPE-L by 8.7 mm on out-of-distribution test content compared to VQ-VAE. Token dimension has a larger impact on performance than quantization levels; FSQ-16-32 (16 levels, 32 dim) achieves 29.7 mm MPJPE-L, substantially lower than FSQ-16-16 (35.7 mm). All encoders maintain success rates above 99.2%, and the human encoder's MPJPE-L is only 0.6 mm higher than the robot encoder's.

The GR00T N1.5 VLA model is evaluated on whole-body loco-manipulation tasks, showing that using FSQ tokens as the action space yields a 42-percentage-point average improvement over explicit SMPL poses, with the benefit most pronounced on complex, long-horizon tasks requiring foot coordination. Motion tokenizer comparisons reveal that FSQ outperforms VQ-VAE, increasing token dimension matters more than adding quantization levels, and cross-encoder alignment between human and robot encoders remains strong. Generalization tests highlight object-specific factors driving success variation, and the model successfully uses feet as manipulators for tasks like opening a pedal-operated trash can.