One-sentence Summary

NVIDIA and ETH Zürich researchers introduce ARDY, a streaming framework for interactive human motion generation that combines a hybrid representation of explicit root features and latent body embeddings with a two-stage autoregressive transformer denoiser to enable real-time control via text prompts and flexible long-horizon kinematic constraints, achieving high fidelity on the HumanML3D and Bones Rigplay benchmarks and being demonstrated in an interactive demo.

Key Contributions

• ARDY is a streaming generation framework for real-time 3D human motion that natively supports online text prompts and flexible, long-horizon kinematic constraints.
• It uses a hybrid motion representation combining explicit root trajectory features with a latent body embedding, balancing precise trajectory control with efficient generative learning.
• A two-stage autoregressive transformer denoiser with variable-length history context handles long-horizon kinematic goals, and evaluations on HumanML3D and the Bones Rigplay dataset confirm high motion quality and constraint adherence, demonstrated by an interactive demo with dynamic text control, keyframe poses, path following, and real-time mouse and keyboard interaction.

Introduction

Generating realistic 3D human motions is critical for character animation, interactive simulations, and humanoid robot control. Offline generative models can produce motions that follow rich text prompts and precise kinematic constraints, but their parallel denoising is too slow for real-time applications. Online autoregressive models are fast, yet they typically sacrifice controllability: methods that support text conditioning lack kinematic control, while those that handle kinematic constraints cannot accept text input, and the few that attempt both suffer from short context windows that limit their ability to understand global text semantics or execute long-horizon kinematic goals. The authors introduce ARDY, an autoregressive diffusion model that leverages a hybrid pose representation decomposing motion into an explicit root trajectory and a latent body embedding. It employs a two-stage interleaved denoiser that first predicts the clean root and then the clean body, enabling streaming generation conditioned on online text prompts and flexible kinematic constraints, such as sparse keyframes, waypoints, and end-effector targets, across long horizons without requiring extra test-time optimization or control modules.

Method

The method ARDY consists of a motion tokenizer and an autoregressive two-stage motion diffusion model. The approach relies on a hybrid motion representation that decouples root motion from body motion. Root trajectories are represented explicitly in global coordinates to facilitate direct control, while body motion is encoded into a compact latent space.

Body Motion Tokenizer To compress high-dimensional explicit body features, the authors train a motion tokenization network.

As shown in the figure above, the tokenizer employs an asymmetric conditional autoencoder architecture. The encoder processes patchified explicit body motion frames and compresses them into latent tokens. These latent body tokens are concatenated with patchified global root motion to form the hybrid motion representation. The decoder then reconstructs the body motion from these hybrid tokens. Notably, the decoder transforms the global root motion into a local representation, consisting of heading angular velocity, linear velocity components, and root height, which serves as conditional input to significantly mitigate foot skating.

Controllable Interactive Motion Generation The generation task is formulated as a conditional autoregressive process. The model synthesizes a short window of future motion tokens starting from the current frame, conditioned on past history, a text prompt, and optional spatial goals. The formulation allows the model to condition on goals beyond the immediate prediction window, enabling long-horizon planning without requiring an additional control policy.

Autoregressive Two-Stage Diffusion Model To solve the generation task, the authors design a transformer-based diffusion denoising model.

The high-level architecture, illustrated in the figure above, processes noisy hybrid motion tokens alongside history tokens, text conditioning, and spatial constraints. Spatial goals are represented as masked explicit motion features. To ensure precise control, the root component of the noisy tokens is overwritten with the root constraints before processing.

The denoiser employs an interleaved two-stage design to sequentially predict root and body motions. First, a Root Transformer predicts the clean global root motion from the noisy inputs. This predicted root motion is then detached and fed into a Body Transformer, which predicts the clean latent body tokens. This decomposition allows the body generation to be conditioned on accurate root trajectories, improving motion fidelity. The final hybrid prediction is decoded to recover the full explicit motion.

Training Process The tokenizer is trained using a reconstruction loss and a specific foot-skating loss that penalizes the velocities of joints predicted to be in contact with the ground: $\mathcal{L}_{\text{skate}} = \frac{\sum_{j \in \mathcal{S}_f} \hat{\mathbf{c}}_j \|\hat{\mathbf{J}}_j\|_2}{\sum_{j \in \mathcal{S}_f} \hat{\mathbf{c}}_j}$Lskate​=∑j∈Sf​​c^j​∑j∈Sf​​c^j​∥J^j​∥2​​ The two-stage denoiser is trained using a modified DDPM framework. The total loss combines a hybrid token loss, a decoded body loss, a goal loss to enforce constraints, and a consistency loss ensuring agreement between predicted joint positions and those derived from predicted rotations via forward kinematics: $\mathcal{L} = \mathcal{L}_{\text{hybrid}} + \mathcal{L}_{\text{dec}} + \mathcal{L}_{\text{goal}} + \mathcal{L}_{\text{consist}}$L=Lhybrid​+Ldec​+Lgoal​+Lconsist​

Test-Time Operation During inference, the model operates autoregressively using a truncated sliding window to manage history and future contexts.

To enable real-time interactivity and handle model inference latency, the authors incorporate a latency-aware replanning mechanism. As depicted in the figure above, when a replan is triggered by new user input or buffer depletion, a buffer of previously generated frames is played back to the user. Simultaneously, these frames serve as history context for an asynchronous generation thread. This strategy effectively masks the inference latency of the diffusion model, ensuring smooth transitions during dynamic replanning.

Experiment

The interactive demo demonstrates ARDY's real-time motion generation with streaming text and spatial constraints, robustly adapting to diverse kinematic goals. Ablation studies on large-scale mocap data validate that the hybrid motion representation, two-stage denoising, and global-to-local root conversion are crucial for motion quality and control accuracy, with hyperparameter analyses revealing stable performance across configurations. Benchmark evaluations show ARDY surpasses prior offline and autoregressive methods, excelling at long-horizon planning and receiving strong human preference in perceptual studies.

Existing real-time 3D motion generation methods sacrifice either spatial control, online text prompting, or long context, while offline methods offer rich spatial control and up to 10-second future horizons but cannot react to dynamic prompts in real time. ARDY is positioned to match the spatial control capabilities of offline models within an interactive, real-time framework. Real-time methods like AMDM and CAMDM lack online text prompting and are restricted to short context windows under 2 seconds, whereas offline counterparts such as MaskControl and Kimodo generate up to 10 seconds of future motion with full spatial control but no real-time capability. MotionStreamer enables real-time generation and online text prompting with long 10-second context, but it does not provide any form of spatial control over root trajectory or joint positions.

Ablation of three architectural designs shows that the hybrid motion representation, global-to-local root conversion, and two-stage denoising each boost motion quality and spatial control. The hybrid representation drastically improves text fidelity and constraint accuracy over an explicit baseline, while the local root conversion cuts skating and the two-stage denoiser yields much lower joint position and waypoint errors. Switching from an explicit to the hybrid representation raises text-to-motion R-precision from 53.90 to 65.47 and reduces FID from 0.065 to 0.027, while waypoint error drops from 0.203 m to 0.024 m and keyframe body error from 0.136 m to 0.023 m. Removing the global-to-local root conversion increases foot skate in text-only generation from 0.264 m/s to 0.303 m/s and degrades all constraint-conditioned metrics, confirming the local representation is essential for physical plausibility. A single-stage denoiser achieves similar text-to-motion quality but struggles with precise control, producing joint position error of 0.101 m and waypoint error of 0.164 m, compared to 0.025 m and 0.024 m for the two-stage design.

An ablation analysis examines generation horizon, diffusion steps, and tokenizer configurations. Longer horizons consistently improve motion fidelity and semantic alignment, while very short horizons lead to instability and poor constraint satisfaction. Moderate horizons best balance spatial control accuracy and responsiveness, and both diffusion steps and tokenizer patch size present trade-offs between quality, detail preservation, and stability. Increasing the generation horizon from 4 to 40 frames steadily raises R-precision and lowers FID, indicating better motion quality and text alignment. The 4-frame horizon yields misleadingly low foot-skating because the model often ignores text prompts, while all constraint errors are substantially higher. Horizons of 8 and 40 frames achieve the lowest joint rotation and position errors in constraint-conditioned generation, with 8-frame models enabling faster transitions between actions. Using only 1 or 2 diffusion steps significantly harms generation quality and constraint adherence, but few-step models still reach competitive performance. A single-frame tokenizer patch causes early training instability and worse final results, whereas a larger patch of 8 frames slightly improves FID and R-precision at the cost of worse skating and constraint accuracy due to lost fine-grained pose details.

ARDY without test-time optimization delivers dramatically better joint position accuracy than the baseline MaskControl model (raw error 4.15 cm vs 46.18 cm), along with lower foot skating and faster inference. While text alignment is slightly lower, the overall motion quality and spatial control are far superior, and the refined MaskControl pipeline achieves high accuracy only at the cost of extremely high latency. Before optimization, ARDY reduces joint position error by more than tenfold compared to MaskControl (4.15 cm vs 46.18 cm) and also improves foot skating and inference speed. ARDY's raw generation maintains competitive text-following while significantly outperforming the specialized MaskControl model in spatial accuracy and motion quality.

ARDY outperforms the autoregressive model DiP on text-conditioned motion generation with sparse joint goals. It delivers better motion quality, text alignment, and constraint accuracy, with substantially lower joint error both when goals are given every 2 seconds and when they are placed far into the future. DiP struggles with long-horizon planning, exhibiting a sharp rise in joint error, while ARDY maintains high accuracy. In the in-horizon setting, ARDY reduces joint error to 2.48 cm, compared to DiP's 9.20 cm, while also improving FID from 0.967 to 0.092. For out-of-horizon goals, DiP's joint error escalates to 17.64 cm, whereas ARDY avoids this degradation and keeps error low, demonstrating effective long-horizon planning.

The evaluation compares ARDY against existing real-time and offline motion generation methods, validating its ability to combine interactive control with strong spatial accuracy. Architectural ablations show that the hybrid representation, local root conversion, and two-stage denoising each significantly improve motion quality, text fidelity, and constraint precision. Additional studies reveal that longer generation horizons consistently boost semantic alignment and motion fidelity, and ARDY outperforms both a specialized offline model and an autoregressive baseline in joint position accuracy and long-horizon planning, maintaining high spatial control without sacrificing real-time responsiveness.