One-sentence Summary

Meta Reality Labs Research introduces LSRM, a Large Sparse Reconstruction Model that scales transformer context windows for feed-forward 3D object reconstruction by adapting native sparse attention through a coarse-to-fine pipeline, 3D-aware spatial routing, and block-aware sequence parallelism, handling $20\times$20× more object tokens and achieving $>2.4$>2.4 dB higher PSNR over prior state-of-the-art methods.

Key Contributions

• This work introduces the Large Sparse Reconstruction Model, a feed-forward 3D reconstruction and inverse-rendering method that scales transformer context windows through native sparse attention, handling 20× more object tokens and more than 2× more image tokens than prior state-of-the-art approaches.
• LSRM adapts sparse attention for 3D reconstruction with an efficient coarse-to-fine pipeline that predicts sparse high-resolution residuals, a 3D-aware spatial routing mechanism that uses explicit geometric distances for 2D-3D correspondences, and a block-aware sequence-parallel strategy with an All-gather-KV protocol for dynamic sparse workloads across GPUs.
• Evaluations on standard novel-view synthesis benchmarks show gains over current state-of-the-art methods of more than 2.4 dB PSNR and more than 40% lower LPIPS, while inverse-rendering experiments show improved texture and geometry details with LPIPS matching or exceeding dense-view optimization.

Introduction

3D foundation models built on transformers have made object-centric reconstruction and inverse rendering much faster, which matters for practical tasks such as creating reliable 3D digital twins. The key limitation is that feed-forward models often produce blurry textures, poor fine detail, and distorted geometry because increasing the number of 3D object and image tokens quickly becomes too expensive in memory and compute. The authors address this by scaling context windows with native sparse attention, using a coarse-to-fine pipeline, 3D-aware block routing, and block-aware sequence parallelism. Their LSRM approach processes far more object and image tokens, improves texture and geometry fidelity, and extends to inverse rendering for relightable 3D assets.

Method

The authors leverage a two-stage transformer pipeline. Stage 1 constructs a coarse dense feature volume from a small set of posed images. Stage 2 initializes from the dense model, expands the input and volume resolutions, and activates only spatially informative image and volume tokens. The sparse stage relies on Native Sparse Attention (NSA), block-aware token partitioning, and a custom sequence-parallelism schedule to scale the context window without destroying spatial locality.

Stage 1 begins with image and volume tokenization. The input consists of $M$M posed images with camera parameters encoded as Plücker rays. Each view is processed into non-overlapping $8 \times 8$8×8 patches, producing an image token map of spatial size $S_{\mathrm{d}}^{\mathrm{img}}=32$Sdimg​=32 for $256 \times 256$256×256 views. For each patch $\mathbf{I}_m$Im​ and its associated Plücker ray $\mathbf{R}_m$Rm​, the model forms a fused image token by adding a linear projection of the raw patch and ray to a projected DINOv3 feature. Because DINOv3 operates on $16 \times 16$16×16 patches, each input patch is first upsampled by $2$2:

This fusion injects pretrained visual structure into the reconstruction backbone and accelerates convergence. The volume tokens $\mathbf{x}$x are initialized from learned 3D positional embeddings. Rather than instantiating a dense parameter volume, the authors factorize the positional embedding along the three axes, learning three independent $1$1D embedding tables $\mathbf{P}^{\mathrm{x}},\mathbf{P}^{\mathrm{y}},\mathbf{P}^{\mathrm{z}}\in\mathbb{R}^{S_{\mathrm{d}}^{\mathrm{vol}}\times d}$Px,Py,Pz∈RSdvol​×d with hidden dimension $d=1024$d=1024 and dense volume resolution $S_{\mathrm{d}}^{\mathrm{vol}}=16$Sdvol​=16. For a token at voxel coordinate $(i,j,k)$(i,j,k), its embedding is the sum

The initialized volume tokens are concatenated with image tokens and processed by dense transformer blocks.

The dense reconstruction transformer uses $24$24 blocks with hidden dimension $1024$1024. It adopts grouped-query attention with $32$32 query heads and only $2$2 key/value heads. This compressed KV layout is important because it reduces attention memory pressure and aligns the architecture with the sparse transformer that follows. Instead of a monolithic self-attention operation, each block uses a gated mixture of cross-attention updates between volume and image tokens. Let $\mathbf{x}$x denote volume tokens and $\mathbf{y}$y denote image tokens. The attention output is

where the gates are produced by linear projections followed by a sigmoid:

This decoupled design allows image-to-volume and volume-to-image attention branches to be adjusted independently in the later sparse stage. The final dense outputs are denoted $\mathbf{x}_{\mathrm{d}}$xd​ and $\mathbf{y}_{\mathrm{d}}$yd​. The volume tokens are projected by a linear layer into a dense feature volume $\mathbf{X}_{\mathrm{d}}$Xd​ with spatial resolution $64$64 and feature dimension $32$32:

This volume can be sampled at a continuous point $\mathbf{p}\in\mathbb{R}^3$p∈R3 with trilinear interpolation, yielding a feature $\mathbf{f}$f that feeds lightweight MLP decoders. Depending on the task, these decoders predict SDF, color, or material properties such as albedo, roughness, and metallic components. For volume rendering, the SDF formulation provides a simple geometric interface for later ray-marching and token filtering.

Stage 2 expands the spatial context substantially. The input image token map can grow from $S_{\mathrm{d}}^{\mathrm{img}}=32$Sdimg​=32 to $S^{\mathrm{img}}=96$Simg=96, and the volume grid can grow from $S_{\mathrm{d}}^{\mathrm{vol}}=16$Sdvol​=16 to $S^{\mathrm{vol}}=96$Svol=96. Processing every token densely would be expensive, so the method first builds a sparse representation by selecting only informative tokens. For image tokens, background regions are discarded using a foreground mask, preserving patches that contain object pixels. For volume tokens, the method uses the geometry encoded by the frozen dense stage. In each voxel, it samples $T=4^3=64$T=43=64 points $\mathcal{P}_{i,j,k}$Pi,j,k​, evaluates their SDF values using the dense feature volume, and marks the voxel as active if it lies close to the surface or is intersected by it:

This filtering step keeps voxels that influence appearance while excluding empty space.

To preserve spatial structure after token pruning, the active tokens are grouped into spatial blocks. Image tokens are grouped in $8\times 8$8×8 blocks, and volume tokens are grouped in $8\times 8\times 8$8×8×8 blocks. This block partitioning is necessary for NSA because naive one-dimensional token splitting would break the locality needed for fine-grained 3D reasoning. Within each block, the token-level keys and values are passed through a residual module and averaged to form compressed key and value features:

These compressed block-level features provide a global coarse context for attention, while selected active blocks provide local fine-grained detail.

The sparse reconstruction transformer keeps the same block count and hidden dimension as the dense model, but replaces dense cross-attention with NSA-based cross-attention branches. It is initialized from the dense weights, which transfers learned geometry and visual correspondence and accelerates convergence. The sparse stage is also trained to predict a residual over the dense prediction rather than rebuilding the full high-resolution volume from scratch. Let $\mathbf{x}_{\mathrm{d}}^{\mathrm{up}}$xdup​ and $\mathbf{y}_{\mathrm{d}}^{\mathrm{up}}$ydup​ denote upscaled informative regions of the dense volume and image outputs. The sparse block update is

for $m=1,\ldots,24$m=1,…,24, followed by the residual combination

The output sparse tokens are projected into a high-resolution sparse feature volume $\mathbf{X}_{\mathrm{s}}$Xs​ with resolution $S_{\mathrm{f}}^{\mathrm{vol}}=384$Sfvol​=384, but only in voxels that remain active. During volume rendering, ray samples inside active surface regions query $\mathbf{X}_{\mathrm{s}}$Xs​ for high-frequency detail, while ray samples in empty or inactive regions query $\mathbf{X}_{\mathrm{d}}$Xd​ for low-frequency structure. Near active-inactive boundaries, the method uses a linear blend between sparse and dense features to avoid discontinuities.

A key module for this sparse stage is 3D-aware block routing. Standard NSA selects KV blocks using attention scores against compressed keys, which can be unreliable in early transformer layers. The visualized routing patterns below show that attention-score-based selection may miss spatially relevant blocks early in the network, while explicit geometry maintains more consistent correspondences.

To make routing geometry-driven, each token is assigned an explicit 3D location. A volume token uses its voxel center. An image token uses the foreground surface point with the highest opacity rendered through the dense feature volume. For query-to-image-block selection, the query point is projected onto each input image plane, nearby image blocks are retrieved per view, and candidates are ranked by their minimal 3D distance to the query point. For query-to-volume-block selection, the method computes 3D Euclidean distances to volume block centers and keeps the nearest blocks. This geometry-based rule improves the correspondence between image evidence and 3D voxel locations. The qualitative ablation below confirms that both the routing mechanism and higher spatial resolutions contribute to sharper texture recovery.

Sparse tokenization also creates a system-level training challenge. Because different instances activate very different numbers of image and volume tokens, worker load varies strongly across GPUs. As shown in the figure below, token count correlates with per-instance computation time, so naive sharding causes straggler effects that stall distributed training.

The authors address this with a custom block-aware sequence parallelism scheme. The goal is to distribute tokens evenly across GPUs while keeping every spatial block intact on a single device. This preserves the block locality required by NSA, allowing window attention and KV-block compression to be executed locally without additional cross-GPU communication for those operations. The flow summarized below shows tokens being redistributed with all-to-all communication before the first NSA block, while spatial blocks remain intact, and queries being returned for decoding and rendering.

To still provide global context during sparse attention, the method uses an all-gather strategy over the compressed and selected KV features. The compressed keys and values are replicated on all devices, while queries remain locally sharded. Because the KV cache is heavily compressed, this gather is relatively inexpensive. The design balances compute across GPUs and enables much larger sparse token counts than standard context-parallel splitting would permit.

The training process is staged to stabilize optimization. The dense model is first trained at lower resolution. Then the dense model is frozen and used to produce token masks, dense residuals, and geometric cues for the sparse model. The sparse stage is first trained at moderate image and volume resolutions to support larger batch sizes and faster convergence, then scaled to high-resolution token grids. At each iteration, the model samples input views and target views from a spherical camera configuration. Training losses supervise rendered color, foreground mask, and depth, with a numerical normal loss added near the end of the schedule to improve geometric fidelity. For inverse rendering, the model initializes from converged novel-view-synthesis weights, encodes background images with an additional linear pathway to reduce lighting material coupling, and initializes the BRDF prediction head from pretrained material weights before fine-tuning under augmented environment maps.

Experiment

The experiments evaluate LSRM on GSO for novel view synthesis and on StanfordORB, DigitalTwinCatalogue, and ObjectsWithLighting for inverse rendering, with ablations isolating context-window scaling and three-dimensional routing. Across these settings, larger context windows, particularly volume-token scaling, and 3D-aware routing improve texture sharpness, facial and geometric structure, and relightability. Pixel-wise inverse-rendering comparisons remain limited by global material ambiguities, but perceptual quality and qualitative detail recovery are substantially better than feed-forward baselines and competitive with optimization methods. The model also generalizes from many-view training to fewer-view evaluation, preserving high-fidelity reconstruction and surpassing prior methods in relighting.

The GSO quantitative results show that LSRM achieves stronger novel-view synthesis quality than the compared feed-forward baselines. Scaling the image and volume token resolutions improves PSNR, SSIM, and LPIPS relative to lower-resolution and dense-representation settings, while 3D-aware block routing provides additional gains. The findings indicate that broader context modeling and geometry-aware attention routing are important contributors to final reconstruction quality. LSRM outperforms Mesh, GS, and LIRM on PSNR, SSIM, and LPIPS for GSO novel-view synthesis. Increasing token resolutions improves novel-view quality compared with smaller-resolution and dense-only variants. 3D-aware routing yields better results than routing omitted at matched token-resolution settings.

The inverse rendering results show that LSRM gives the strongest perceptual quality across StanfordORB, DTC, and OWL, achieving the lowest LPIPS on all three benchmarks. Pixelwise fidelity improves moderately, while texture detail and relightability benefit from more consistent high-frequency reconstruction. The results also indicate better geometric consistency, especially the marked reduction in Chamfer distance on StanfordORB. LSRM consistently achieves the lowest LPIPS across all inverse rendering datasets. Perceptual gains are larger than PSNR gains, which are limited by global material ambiguities. LSRM shows the best StanfordORB Chamfer distance among the compared methods. Qualitative observations support recovery of fine textures, legible text, and small geometry details.

The reduced input view results on the GSO benchmark show that LSRM retains strong novel view synthesis quality when using eight views, even though it is trained with twelve to sixteen views. Its reported metrics remain higher for PSNR and SSIM and lower for LPIPS than LIRM for both eight and sixteen input views. The comparison suggests that scaling context tokens supports strong generalization to fewer views without fine tuning for a smaller view count. Eight view LSRM outperforms LIRM in both view settings on the reported image quality metrics. Fewer input views lead to only slight texture sharpness loss while preserving a clear advantage over the prior feed forward baseline.

When inverse rendering uses only six input views, the proposed model gives the best reported quality across the three tested benchmarks, beating RelitLRM and LIRM on appearance, perceptual similarity, and geometric error. The advantage is notable because the model is trained with 12 to 16 views and not fine-tuned for reduced-view inference. Increasing the number of views to 18 further improves several reconstruction and geometry metrics, though some material and perceptual scores can vary. Achieves the strongest six-view inverse rendering results on all three benchmarks. Shows lower geometric error than competing methods on StanfordORB. Generalizes well below its training view count of 12 to 16 inputs.

Experiments evaluate novel-view synthesis on GSO and inverse rendering across StanfordORB, DTC, and OWL, showing that LSRM consistently outperforms feed-forward baselines in perceptual quality and overall reconstruction fidelity. Increasing token and volume resolutions, along with 3D-aware routing, strengthens novel-view results, while inverse rendering experiments reveal the largest gains in perceptual consistency, texture detail, and geometry, especially on StanfordORB. The model also generalizes well to fewer input views than used in training, maintaining clear advantages over prior methods with only limited degradation in texture sharpness or geometry.