One-sentence Summary

4DAnyone, introduced by researchers from Zhejiang University et al., reconstructs 4D humans from an uncalibrated monocular video by generating multiview-consistent videos and lifting them into 4D Gaussian Splatting, with Reference Context Packing compressing reference views to a fixed-length context and Target Context Routing rotating target-view groupings to overcome bounded-attention-context failures, outperforming prior methods on DNA-Rendering and DyMVHumans.

Key Contributions

• The paper introduces 4DAnyone, a generation-then-reconstruction framework that uses sparse 3D skeletons as explicit geometric conditioning to synthesize reconstruction-grade multiview-consistent videos from an uncalibrated monocular video and lift them into 4D Gaussian Splatting for human reconstruction.
• Reference Context Packing compresses growing reference views into a fixed-length mixed-resolution context with O(1) reference-context complexity, while Target Context Routing rotates target-view groupings during denoising to share context across groups at high-noise steps and stabilize details at low-noise steps.
• The MVGameHuman dataset is built with an in-house game engine and combined with light-stage and in-the-wild video datasets for training. Experiments on DNA-Rendering and DyMVHumans show that 4DAnyone outperforms prior methods in both novel-view video quality and downstream 4DGS reconstruction, with robust in-the-wild generalization.

Introduction

Reconstructing 4D humans that can be rendered from arbitrary viewpoints is important for embodied AI, immersive content creation, and virtual reality. Recent 4D Gaussian splatting methods enable real-time photorealistic dynamic rendering, but they still require dense multi-view video from calibrated static camera arrays. Prior attempts using monocular video or camera-controlled video diffusion struggle with unknown camera parameters, occluded body regions, high-fidelity appearance, and especially cross-view consistency when generating enough views for reconstruction. The authors introduce 4DAnyone, a generation-assisted framework that uses 3D skeleton conditioning, Reference Context Packing to keep appearance guidance at constant context size, and Target Context Routing to share global structure across target-view groups during denoising, enabling reconstruction-grade multi-view video from an uncalibrated monocular video.

Method

The authors propose 4DAnyone, a framework that generates reconstruction-grade multi-view human videos from a single monocular source video with unknown camera intrinsics and poses. These generated videos are subsequently used to train a high-fidelity 4D Gaussian Splatting model.

As shown in the figure below:

The pipeline begins by leveraging a Human Mesh Recovery model to estimate a 3D skeleton sequence from the source video. This sequence is rendered into depth-buffered skeleton videos at prescribed static target viewpoints. A 3D-aware skeleton encoder then injects these geometric cues into the noisy target latents. The Diffusion Transformer denoises the latents conditioned on the source video, the skeleton signals, and a Reference Context Packing mechanism. To handle the generation of tens of views efficiently, Target Context Routing exchanges context among target view groups, reducing structural drift under a fixed per-group memory budget.

To provide reliable geometric guidance without the noise associated with dense signals like depth maps, the authors prioritize accuracy over density by using 3D skeletons. To resolve pose ambiguity inherent in 2D renderings, such as front-back occlusions, the 3D skeleton is rasterized with a pixelwise z-buffer. This produces an occlusion-aware rendering without requiring extra input channels. For keypoint selection, a compact 40-keypoint subset is utilized, retaining body, foot, and palm-level hand keypoints along with auxiliary neck, shoulder, and elbow landmarks. Facial and fine finger keypoints are excluded to prevent artifacts from noisy detections, allowing the model to learn these details directly from the source video reference. The skeleton encoder $g_\phi$gϕ​ injects the skeleton condition into the noisy latent tokens as a residual:

where $\mathbf{S}_i$Si​ denotes the depth-buffered skeleton video for target view $i$i, and $\mathbf{z}_i^t$zit​ denotes the noisy latent tokens at timestep $t$t. The final projection layer of the encoder is zero-initialized to ensure stable training from pretrained weights.

Generating high-fidelity 4D reconstructions requires tens of target-view videos, but joint denoising is computationally prohibitive. Splitting views into groups introduces consistency bottlenecks regarding both appearance conditioning and global structural drift. The authors address these through Reference Context Packing and Target Context Routing.

As shown in the figure below:

Reference Context Packing supplies scalable appearance context. As earlier generated views serve as references for later rounds, cross-view redundancy grows. Instead of appending every generated view as full context, the authors use multi-scale patchify layers to pack the growing set of references into a fixed set of token slots. Using a patchify layer $\mathcal{P}_r$Pr​ with kernel and stride $(1, 2r, 2r)$(1,2r,2r) for compression ratio $r$r, the system produces $\frac{1}{r^2}$r21​ as many tokens as the standard layer. The packed reference tokens are assembled along spatial dimensions and concatenated with source and target tokens along the view dimension. The fixed context is formulated as:

During inference, reference views are generated in two rounds using farthest-point sampling to maximize viewpoint coverage. Round 1 generates four reference videos conditioned only on the source, and Round 2 generates four additional videos conditioned on the source and a subset of the first round. These are then packed into the fixed context for generating all remaining target views in four-view groups.

Target Context Routing addresses the generation-side bottleneck where independently denoised groups might drift structurally. The authors observe that global structure is established at high noise levels. Consequently, inference is divided into two phases based on a switching timestep $t_s$ts​. In the high-noise phase ($t > t_s$t>ts​), the ordered view indices are cyclically shifted by the step index and repartitioned into four-view groups at each step, allowing views to exchange context and propagate global structure. In the low-noise phase ($t \leq t_s$t≤ts​), adjacent four-view groups are fixed so neighboring views can jointly refine details, stabilizing appearance and cross-view transitions.

The training protocol follows a three-stage curriculum. Stage 1 trains on foreground-only videos to learn skeleton-conditioned camera control. To decouple pose from appearance, source and target view clips are sampled from different temporal windows with a 20% probability. Stage 2 extends training to all multi-view datasets without foreground masking, which prevents edge noise from mask boundaries and allows the model to learn lighting and shadow cues from backgrounds. Stage 3 incorporates monocular datasets to improve in-the-wild generalization, during which finger keypoints are removed from the skeleton input to force the model to infer hand details from the source video. The model is trained using a combined loss function:

where $\mathcal{L}_{\mathrm{latent}}$Llatent​ is the standard MSE flow-matching loss in latent space, and $\mathcal{L}_{\mathrm{LPIPS}}$LLPIPS​ is a perceptual reconstruction loss computed on decoded frames with $\lambda = 0.25$λ=0.25 to mitigate artifacts from high spatial compression. To reduce GPU memory consumption, the perceptual loss is evaluated on body-part-aware semantic crops.

Experiment

Across DNA-Rendering and DyMVHumans test scenes, 4DAnyone is evaluated by generating 16 target views from a single source video, with metrics covering 4DGS reconstruction, generated video consistency, and reconstruction quality. Qualitative comparisons show that the method maintains geometric accuracy and coherent cross-view appearance, whereas baseline models suffer from side/back distortion, depth errors, or camera misalignment. Ablations confirm that Reference Context Packing and Target Context Routing complement each other to improve multi-view consistency, and depth‑buffered 3D skeleton conditioning resolves front‑back ambiguity; further experiments on in‑the‑wild videos demonstrate strong generalization to challenging poses, motions, and complex backgrounds.

The training set combines multi-view and monocular human-centric video. Multi-view sources include synthetic game-engine footage, varied-background clips, and real captures, with DNA-Rendering leading in video count and camera count among multi-view sources. Monocular TedTalk and Pexels add single-camera variety, with Pexels contributing the largest actor pool for in-the-wild generalization. Multi-view sources range from synthetic game-engine captures to real recordings, with DNA-Rendering offering the largest video count and highest camera count among multi-view sources. Monocular sources add single-camera video and broad actor diversity; Pexels contributes the largest actor count and highest resolution among the listed datasets.

The proposed method achieves the strongest results across generated-video consistency, 4DGS reconstruction, and generated-video reconstruction on DNA-Rendering and DyMVHumans. TrajectoryCrafter shows the weakest performance, especially on DNA-Rendering, with failures attributed to depth error accumulation and front-to-back changes. A fine-tuned ReCamMaster is the closest baseline for video consistency but still trails in reconstruction quality due to imprecise camera control. The proposed method outperforms all baselines across all three evaluation dimensions on both datasets. TrajectoryCrafter records the lowest scores and suffers from severe geometry distortion and front-to-back failure, particularly on DNA-Rendering. A fine-tuned ReCamMaster achieves the best baseline consistency scores but still lags in reconstruction fidelity due to inaccurate camera control.

This ablation study measures generated video consistency across configurations that remove reference context packing and target context routing. Removing either component lowers PSNR and SSIM while raising LPIPS, and removing both produces the largest degradation. Among routing strategies, sliding grouping is the only one that improves all three metrics over fixed grouping. Removing both reference context packing and target context routing leads to the lowest PSNR and SSIM and the highest LPIPS among the evaluated configurations. Reference context packing contributes more than target context routing in this setting: removing RCP alone shows larger metric degradation than removing TCR alone. Sliding routing improves all consistency metrics relative to fixed grouping, while random routing provides no measurable gain and strided routing degrades metrics.

The evaluation uses a training set combining multi-view and monocular human-centric videos, with DNA-Rendering providing the largest multi-view coverage and Pexels contributing the widest actor diversity for in-the-wild generalization. The proposed method outperforms all baselines on video consistency, 4DGS reconstruction, and generated-video reconstruction across both DNA-Rendering and DyMVHumans, while TrajectoryCrafter suffers from depth errors and a fine-tuned ReCamMaster is limited by imprecise camera control. Ablation results confirm that removing reference context packing or target context routing degrades consistency, with reference context packing contributing more, and that sliding grouping improves all metrics over fixed grouping.