One-sentence Summary

Nanyang Technological University, Kuaishou Technology, and The Hong Kong University of Science and Technology (Guangzhou) present MetaView, a diffusion-based monocular novel view synthesis framework that integrates implicit geometry priors from a feed-forward geometry perception network with metric depth to achieve geometry consistency and precise camera control under large viewpoint changes, significantly outperforming existing methods.

Key Contributions

• MetaView combines implicit geometry priors from a feed-forward network (DepthAnything3) with metric depth cues to achieve structural consistency and precise camera control under large viewpoint changes, without explicit 3D reconstruction.
• The framework augments a pre-trained text-to-image diffusion model (Qwen-Image-Edit) by concatenating reference image tokens and injecting depth-derived spatial signals, enabling geometry-aware novel view synthesis.
• Experiments across multiple datasets show that MetaView significantly outperforms existing methods in monocular large-viewpoint scenarios and generalizes robustly to out-of-domain data.

Introduction

The authors address monocular novel view synthesis, a task that synthesizes target views from a single image under arbitrary camera poses, which is critical for scalable 3D perception and generation. Existing diffusion-based methods either rely on explicit sparse reconstruction that constrains generalization, or learn 3D structure purely from pixel supervision, leading to scale drifting and spatial inconsistency under large viewpoint changes. The authors propose MetaView, a framework that enhances spatial awareness without explicit reconstruction by injecting implicit geometry priors from a feed-forward depth network and minimal 3D cues into a pretrained multi-modal diffusion transformer via non-invasive parallel attention layers, enabling precise camera control and robust synthesis.

Method

The authors tackle novel view synthesis from a single source image and a target camera pose. The method builds on a pretrained text-to-image diffusion model and injects both explicit 3D cues and implicit geometry priors while keeping the original backbone frozen. The overall pipeline takes a source image $X^{src}$Xsrc, its camera intrinsics $K$K, and a relative target pose $T \in SE(3)$T∈SE(3), and generates the corresponding novel view $X^{gen}$Xgen under flow matching.

The base model is Qwen-Image-Edit, a multi-modal DiT (MM-DiT) trained with flow matching. It processes image tokens $X = [X^{src} : X^{gen}]$X=[Xsrc:Xgen] and text tokens $C$C through stacked multi-modal attention blocks. In each block, modality-specific queries, keys, and values are concatenated and fed into a self-attention operation that enables cross-modal fusion. Positional information is encoded via Rotary Positional Encoding (RoPE): text tokens use 1D RoPE, while image tokens use 2D RoPE on grid coordinates $(i, j)$(i,j) and an image index $k$k.

To eliminate scale ambiguity between the camera pose and the scene geometry implicitly learned by the diffusion model, the authors introduce explicit 3D cues. They estimate metric depth $z$z from the source image using an off-the-shelf metric depth estimator (DepthAnything3-Metric). The camera intrinsics and extrinsics are encoded into RoPE following a frustum-aware formulation. Specifically, the projection matrix $\tilde{P} = [K \ \mathbf{0}^{3 \times 1}] T$P~=[K 03×1]T is used to construct a block-diagonal rotation matrix $D_t^{\text{RoPE}}$DtRoPE​ that combines a projection-derived part and a grid part. The grid part extends the usual 2D RoPE with a third axis for the downsampled depth $z'$z′, so each token’s position is represented by $(i, j, z')$(i,j,z′). The resulting RoPE transformation is applied to the attention queries, keys, and values via matrix products, aligning the model’s internal spatial perception with the camera geometry. This injection provides an explicit metric scale without handcrafted reconstruction operations such as depth lifting.

Beyond explicit scale, relative geometric relationships are crucial for view synthesis. The authors extract implicit relative geometry priors from the same depth estimation network. Intermediate features from selected decoder layers of DepthAnything3 are concatenated channel-wise and linearly projected to form geometry tokens $G \in \mathbb{R}^{L \times d}$G∈RL×d. These tokens encode fine-grained scene structure in a tokenized form compatible with the DiT architecture. In the attention layers, $G$G is concatenated with the image tokens $X$X, and all tokens share the same RoPE formulation. The geometry tokens and source image tokens are assigned the identity camera pose $T^o = \mathbf{I}$To=I and the downsampled depth $z'$z′, while the generated image tokens receive the target pose $T$T and a zero depth component. This design allows $X^{gen}$Xgen to attend to geometry-aware guidance without directly constraining the generated depth.

To integrate these geometric signals while preserving the rich semantic knowledge of the pretrained model, the authors adopt a parallel attention adaptation strategy. All pretrained parameters of the MM-DiT backbone are frozen. A new geometry stream is introduced alongside the original image and text streams. In each transformer block, a parallel image–geometry attention layer is added that models interactions between $X$X and $G$G. Its output is summed with the original image–text attention output. Dedicated projection parameters are introduced for the image–geometry attention, while the geometry tokens and image tokens share the same feed-forward network. A linear projection $W_G$WG​ maps the geometry tokens to the same dimension as image tokens before they enter the network. Only these newly added parameters are trained, using the flow matching loss:

This design efficiently injects both explicit scale cues and implicit relative geometry priors into the generation process, enabling stable novel view synthesis without sacrificing the generalization capability of the pretrained diffusion model.

Experiment

MetaView is evaluated on DL3DV, RealEstate10K, and Sekai-Real-Walking–HQ datasets with difficulty levels based on view overlap, comparing against both reconstruction-based and implicit baselines. Qualitative results demonstrate that MetaView produces coherent novel views with precise camera control under arbitrary viewpoint changes, while baselines exhibit scale drift, blurring, or distortions, especially under large viewpoint variations. Quantitative evaluation and the proposed DMD metric confirm MetaView's superior performance and robustness across all difficulties, with DMD better reflecting perceptual quality than low-level metrics. Ablations highlight the importance of geometry tokens and z-axis conditioning for accurate spatial reasoning, and the method shows strong generalization to diverse out-of-domain scenes.

On the DL3DV dataset, reconstruction-based methods perform well on easy views but degrade sharply as difficulty increases, while implicit methods struggle across all levels due to imprecise viewpoint control. MetaView achieves the best results overall, and the DMD metric more clearly separates methods on hard views where PSNR and SSIM lose discriminative power. Reconstruction-based methods like ViewCrafter and Gen3C show strong PSNR and SSIM on easy scenes but suffer large drops on hard scenes, while implicit methods such as HY-World-1.5 and Lingbot-World consistently underperform across all difficulties. On the hard subset, PSNR and SSIM fail to reflect perceptual quality, but the DMD metric reveals a clear performance gap, with MetaView outperforming all compared methods by a substantial margin.

Gen3C achieves the best performance on DL3DV and Sekai across all metrics, while on RealEstate10K it leads in perceptual metrics LPIPS and DMD, with PSNR and SSIM closely contested by Voyager and PE-Field. The DMD metric exposes wider performance gaps, particularly on DL3DV where Gen3C's DMD of 20.01 is markedly lower than the next best method. Gen3C outperforms all compared methods on DL3DV and Sekai datasets across PSNR, SSIM, LPIPS, and DMD. On RealEstate10K, Gen3C achieves the lowest LPIPS (0.2306) and DMD (9.01), while Voyager leads PSNR and PE-Field leads SSIM.

The full model achieves the highest scores across all metrics. Removing geometry tokens or the z-axis consistently degrades performance, with the z-axis ablation causing the largest drops in PSNR and SSIM. When adapted to a FLUX.1-Kontext backbone, the method outperforms the PE-Field baseline, demonstrating cross-model robustness. Ablating geometry tokens lowers PSNR and SSIM while increasing LPIPS and DMD, reflecting weaker fine-grained structural reasoning. Removing the z-axis leads to the lowest PSNR and SSIM among all variants, confirming that scale cues are essential for accurate camera control.

The experiments evaluate view synthesis methods on DL3DV, Sekai, and RealEstate10K datasets, examining performance across easy and hard viewpoints and validating the proposed model's components. Reconstruction-based methods degrade sharply on hard views, while implicit methods underperform due to imprecise viewpoint control; MetaView and Gen3C achieve the best overall results, and the DMD metric reveals perceptual differences that PSNR and SSIM miss. Ablation studies demonstrate that geometry tokens and z-axis scale cues are essential for accurate camera control, and the approach remains robust when adapted to a different backbone.