One-sentence Summary

Researchers from Alaya Lab and The University of Tokyo introduce WorldSculpt, which adapts the single-object $3\mathrm{D}$3D generative prior Pixal3D with a multi-view conditioning pathway to compositionally reconstruct cluttered scenes containing hundreds of objects without any scene-level training, and they present UE-MeshyScene, a photorealistic benchmark with per-object annotations and ground-truth meshes, demonstrating superior performance and scalability across single-object, controlled multi-object, and dense-scene evaluations.

Key Contributions

• Introduces WorldSculpt, a framework that adapts a single-object 3D generative prior with a multi-view conditioning pathway to generate compositional mesh scenes from posed observations, enabling plausible completion of occluded geometry without scene-level training.
• Presents UE-MeshyScene, a photorealistic benchmark of densely cluttered scenes with 93 to 701 objects per scene, per-object annotations, and ground-truth meshes, addressing the scarcity of evaluation data for compositional scene generation.
• Demonstrates consistent outperformance over prior baselines across single-object, controlled multi-object, and UE-MeshyScene evaluations, with the largest gains under dense clutter and severe occlusion, and shows applicability by converting existing 3DGS worlds into compositional mesh scenes.

Introduction

Recent generative world models can synthesize persistent 3D environments from a single image or text prompt, but they typically produce a unified scene representation, such as a fused mesh or a set of Gaussians, without separating individual objects. This prevents downstream applications like gaming, AR/VR, simulation, and robotics from selecting, moving, or re-simulating objects independently, as they require scenes decomposed into distinct, usable assets. Prior work either reconstructs scenes as monolithic geometry with incomplete occluded regions or generates individual objects from a single pose-free image, lacking multi-view conditioning and scene-coordinate grounding, while existing compositional methods are limited to simple, single-image scenes.

The authors introduce WorldSculpt, a framework that adapts a strong object-level generative prior, using Pixal3D as a case study, to handle complex scenes with hundreds of densely occluded objects. They extend the prior with a multi-view conditioning pathway that maps each object's observations into an anchor-aligned canonical frame, fusing DINOv3 features from multiple views via a permutation-invariant aggregator. The frozen prior is adapted with zero-initialized projection layers and low-rank adapters, enabling plausible completion of unobserved geometry while grounding shapes in visible evidence, all without scene-level training. Each object is generated as an individual mesh and placed into the shared world frame via canonical-to-world transformations. To evaluate under genuine clutter, they also introduce UE-MeshyScene, a photorealistic benchmark with up to 701 objects per scene and exact per-object ground truth, showing consistent improvements over baselines, especially under severe occlusion.

Dataset

The authors evaluate their method across three increasingly difficult settings, each supported by a dedicated dataset. The datasets are used for controlled single-object generation, compositional scene generation, and large-scale generation in densely cluttered environments.

Dataset composition and sources

• Toys4k: A collection of clean, isolated object models used for controlled single-object evaluation. It contains no scene-level factors, allowing the authors to isolate the model's ability to recover object geometry from partial multi-view observations.
• Toys4k-Scene: A synthetic benchmark built by placing multiple Toys4k objects into cluttered layouts and rendering short orbiting sequences. It preserves the clean ground-truth geometry of Toys4k while introducing substantial inter-object occlusion and challenging multi-object configurations.
• HouseCat6D: A real-world dataset with captures of tabletop scenes and scanned ground-truth meshes for individual objects. Its layouts are relatively sparse, objects are mostly simple household items, and inter-object occlusion is generally mild.
• UE-MeshyScene: A large-scale benchmark authored and rendered in Unreal Engine 5.8, consisting of six environments: an aircraft hangar, an abandoned-city interior, an old cathedral, an architectural-visualization office, a Japanese school, and a desert town.

Key details for each subset

• Toys4k: The authors vary the number of input views from 1 to 16 and the per-view occlusion fraction from 0 to 75 percent, measuring how generation quality changes as observations become fewer and more incomplete.
• Toys4k-Scene: Many objects contain thin or intricate structures and are densely arranged, leaving only partial observations from typical viewpoints. This setting is particularly challenging for methods that rely on a single image.
• HouseCat6D: Used to evaluate how well a model trained on synthetic renders transfers to real imagery. The layouts are sparse, and inter-object occlusion is mild compared with Toys4k-Scene.
• UE-MeshyScene: The scenes contain between 93 and 701 objects each, with 2,299 objects overall. Each scene is rendered along a smooth camera trajectory at a resolution of 2560 by 1440, producing 265 to 1758 views per scene and 5,964 views in total. For every frame, the dataset provides an RGB image, camera pose, per-object instance masks, per-object 3D bounding boxes, and a metric depth map. The authors release the full set of annotations, although depth is not used by their method.

How the data is used

• The authors evaluate canonical-space generation of individual objects on Toys4k, isolating the ability to recover object geometry from partial multi-view observations.
• They evaluate compositional scene generation on Toys4k-Scene and HouseCat6D.
• They evaluate large-scale compositional generation on UE-MeshyScene, where scenes contain hundreds of objects under severe mutual occlusion.
• For UE-MeshyScene, every asset is a known 3D model placed with a known transformation, providing complete per-object mesh ground truth, exact camera poses, and 3D bounding boxes. This enables direct evaluation in a common world coordinate frame while retaining the complexity of densely cluttered environments.

Metrics and evaluation protocol

• For each object, the authors decode the prediction into a mesh and compare it with the corresponding ground-truth geometry using Chamfer distance with both L2 and L1 distances, Earth Mover's Distance, and F-Score.
• For single-object evaluation in canonical space, both prediction and ground truth are normalized with respect to the ground-truth unit sphere. Since different baselines adopt different canonical coordinate conventions, the authors perform per-instance ICP alignment before computing metrics.
• For compositional scene evaluation, each generated object is transformed from its canonical frame into the world frame using the canonical-to-world transformation. They compare directly with the ground-truth mesh in the shared world frame without per-object ICP alignment. Distance-based metrics are normalized by the diagonal length of the ground-truth bounding box for each object, evaluating both object geometry and its placement within the composed scene.

Method

Given a set of $N$N posed images $\{ I _ { n } \} _ { n = 1 } ^ { N }${In​}n=1N​ with known camera intrinsics $\{ K _ { n } \} _ { n = 1 } ^ { N }${Kn​}n=1N​ and extrinsics $\{ T _ { n } ^ { \mathrm { c w } } \} _ { n = 1 } ^ { N }${Tncw​}n=1N​, along with per-view instance masks $S _ { k n }$Skn​ and coarse world-space localization boxes $B _ { k } ^ { \mathrm { l o c } }$Bkloc​ for each object $k$k, the goal is to produce a compositional scene representation:

$$\mathcal { M } = \{(\mathcal { M } _ { k } ^ { \mathrm { c } }, T _ { k } ^ { \mathrm { o w } }) \} _ { k = 1 } ^ { K }$$M={(Mkc​,Tkow​)}k=1K​

where $\mathcal { M } _ { k } ^ { \mathrm { c } }$Mkc​ is an individual mesh in the canonical frame of object $k$k, and $T _ { k } ^ { \mathrm { o w } }$Tkow​ maps that frame to the world coordinate system. The method consists of three main steps: constructing an anchor-aligned virtual canonical frame, generating an individual object mesh with a multi-view conditioned 3D generative prior, and transforming the mesh back into the world frame.

Refer to the framework diagram:

The coarse localization box $B _ { k } ^ { \mathrm { l o c } }$Bkloc​ is not directly used as the generation volume because its side lengths are unequal and its orientation may not align with the camera-relative canonical orientation expected by the pretrained object prior. Instead, the authors construct an anchor-aligned virtual canonical cube. For each object, they select an anchor view $a _ { k }$ak​ (the view where the object is most fully observed at inference, or a random view during training). The anchor camera orientation induces a rotation $R _ { k } \in \mathsf { S O } ( 3 )$Rk​∈SO(3) mapping the anchor view to the canonical front-view direction. Using the normalized cube $\mathcal { V } = [ - \frac { 1 } { 2 }, \frac { 1 } { 2 } ] ^ { 3 }$V=[−21​,21​]3 as the canonical spatial domain, the center $c _ { k }$ck​ of $B _ { k } ^ { \mathrm { l o c } }$Bkloc​, and its largest side length $s _ { k }$sk​, the canonical-to-world transformation is defined as:

$$T _ { k } ^ { \mathrm { o w } } = \left[ \begin{array} { c c } s _ { k } R _ { k } & c _ { k } \\ 0 & 1 \end{array} \right]$$Tkow​=[sk​Rk​0​ck​1​]

This similarity transformation preserves object proportions. To handle inaccurate localization boxes, $s _ { k }$sk​ is increased until the projected cube covers the object masks in all selected views.

For each selected view $n$n, the anchor-aligned cube is projected into the image, cropped to its extent, background-masked using $S _ { k n }$Skn​, and resized to the generative model input resolution to produce the object-centric image $\bar { I } _ { k n }$Iˉkn​. The adjusted camera intrinsics are $\bar { K } _ { k n } = A _ { k n } K _ { n }$Kˉkn​=Akn​Kn​, where $A _ { k n }$Akn​ is the crop-and-resize transformation. A canonical voxel $x$x is projected into the cropped observation via:

$$\pi _ { k n } ( x ) = \Pi \left( \bar { K } _ { k n } \left[ \begin{array} { c c } I _ { 3 } & 0 \end{array} \right] ( T _ { n } ^ { \mathrm { c w } } ) ^ { - 1 } T _ { k } ^ { \mathrm { o w } } \tilde { x } \right)$$πkn​(x)=Π(Kˉkn​[I3​​0​](Tncw​)−1Tkow​x~)

where $\tilde { x } = [ x ^ { \top }, 1 ] ^ { \top }$x~=[x⊤,1]⊤ and $\Pi$Π denotes perspective division.

The per-object generator $\Phi$Φ is instantiated from Pixal3D, a native-3D generative model. The authors use its first two flow-matching stages (sparse structure and shape) to determine geometry. To ground the generation in the anchor-aligned canonical volume using multiple posed observations, they introduce a multi-view conditioning pathway. Each canonical observation $\bar { I } _ { k n }$Iˉkn​ is encoded with DINOv3 to produce a dense feature map $F _ { k n } = E _ { \mathrm { DINO } } ( \bar { I } _ { k n } )$Fkn​=EDINO​(Iˉkn​). For each voxel $x$x, the feature from view $n$n is sampled at its projected location $g _ { n } ( x ) = F _ { k n } ( \pi _ { k n } ( x ) )$gn​(x)=Fkn​(πkn​(x)), lifting 2D features into a shared 3D feature grid.

To aggregate these lifted features, an IBRNet-style permutation-invariant module is used. For each voxel $x$x, the cross-view mean $\mu ( x )$μ(x) and variance $\sigma ^ { 2 } ( x )$σ2(x) are computed. Two lightweight MLPs refine the features and predict aggregation logits:

$$g _ { n } ^ { \prime } ( x ) = \mathrm { MLP } _ { \mathrm { feat } } ( [ g _ { n } ( x ), \mu ( x ), \sigma ^ { 2 } ( x ) ] )$$gn′​(x)=MLPfeat​([gn​(x),μ(x),σ2(x)]) $$w _ { n } ( x ) = \mathrm { MLP } _ { \mathrm { weight } } ( [ g _ { n } ( x ), \mu ( x ), \sigma ^ { 2 } ( x ) ] )$$wn​(x)=MLPweight​([gn​(x),μ(x),σ2(x)])

The final voxel feature is the cross-view mean plus a softmax-weighted residual:

$$g _ { \mathrm { out } } ( x ) = \mu ( x ) + \sum _ { n \in \mathcal { J } _ { k } } \alpha _ { n } ( x ) g _ { n } ^ { \prime } ( x ), \quad \alpha _ { n } ( x ) = \frac { \exp ( w _ { n } ( x ) ) } { \sum _ { m \in \mathcal { J } _ { k } } \exp ( w _ { m } ( x ) ) }$$gout​(x)=μ(x)+n∈Jk​∑​αn​(x)gn′​(x),αn​(x)=∑m∈Jk​​exp(wm​(x))exp(wn​(x))​

The final layer of $\mathrm { MLP } _ { \mathrm { feat } }$MLPfeat​ is zero-initialized, so the aggregator starts as an exact cross-view mean. The resulting 3D conditioning grid $G _ { k }$Gk​ is injected into both Pixal3D geometry stages. Global image tokens are extracted only from the anchor observation to preserve the pretrained single-image convention. The aggregated condition is aligned with 3D token locations, projected, and added to block features. The original Pixal3D parameters are frozen, while attention and projection layers are adapted using LoRA.

During training, the sparse-structure and shape stages are trained independently. For each object, a variable number of conditioning views are sampled, and one is randomly chosen as the anchor. Only the multi-view aggregator, conditioning injection layers, and LoRA parameters are optimized. To bridge the gap between clean training renders and cluttered scene observations, a conditioning-view augmentation curriculum is applied. This includes simulating occlusions via 2D masks or 3D-consistent occluders, perturbing non-anchor camera poses, degrading segmentation boundaries, and downsampling observations. The augmentation strength gradually increases during the first 3k iterations.

At inference, the anchor is chosen as the view with the most fully observed object. The canonical cube is constructed, and a bounded number of conditioning views are selected. The object is generated by running the two Pixal3D geometry stages, and the resulting canonical mesh $\mathcal { M } _ { k } ^ { \mathrm { c } }$Mkc​ is placed into the scene using $\mathcal { M } _ { k } ^ { \mathrm { w } } = T _ { k } ^ { \mathrm { o w } } ( \mathcal { M } _ { k } ^ { \mathrm { c } } )$Mkw​=Tkow​(Mkc​).

As shown in the figure below:

This pipeline can also be applied to convert generated 3D Gaussian Splatting worlds into compositional mesh scenes by rendering posed observations, recovering object masks and coarse 3D localizations, and applying the standard inference pipeline without additional training.

Experiment

The evaluation spans three increasing difficulty levels: canonical-space single-object generation on Toys4k, compositional scene generation on synthetic Toys4k-Scene and real-world HouseCat6D, and large-scale cluttered scenes on the newly introduced UE-MeshyScene benchmark. The method remains competitive with single-view baselines when given one view but consistently outperforms them as input views increase, with multi-view conditioning providing strong robustness to partial observations and occlusion. In compositional settings, the approach recovers more detailed geometry and more accurate world-frame placement than baselines, with gains most pronounced on heavily occluded scenes. An ablation shows that the learned multi-view fusion module outperforms simple mean averaging, with its advantage growing as scene clutter and occlusion increase, and a proof-of-concept demonstrates converting a generated 3DGS world into compositional meshes without retraining.

UE-MeshyScene is a large-scale synthetic benchmark rendered in Unreal Engine 5.8 at 2560x1440, containing 2,299 objects across six scenes with 5,964 total rendered views. The scenes vary widely in object count and view coverage, with object counts ranging from 93 to 701 and view counts from 265 to 1,758, reflecting diverse levels of complexity and occlusion. The dataset includes six distinct scenes, with object counts ranging from 93 in Abandoned City to 701 in Desert Town. View counts per scene vary from 265 in Abandoned City to 1,758 in Japanese School, indicating different observation densities. The largest scenes by object count are Desert Town (701) and Office (678), while the smallest is Abandoned City (93).

The table evaluates canonical-space 3D object generation on Toys4k under varying input view counts and per-view occlusion levels, comparing the proposed model against single-view baselines and multi-view variants. Results show that the proposed model remains competitive at one view and improves consistently with more views, especially under heavy occlusion, where additional views stabilize distance errors and F-scores. The learned multi-view fusion aggregator generally outperforms simple mean averaging, with the gap widening as views and occlusion increase. With clean inputs, the model matches a dedicated single-view pipeline at one view and surpasses all baselines as the number of views increases. Performance improves consistently with more input views across all metrics. Under occlusion, additional views substantially mitigate degradation; with 16 views, metrics remain stable up to 50% per-view occlusion. The learned view-weighting aggregator performs comparably to mean fusion at one view but shows clear gains at higher view counts and occlusion levels. The benefit of learned fusion is most pronounced in cluttered scenes with many mutually occluding objects, where it improves all reported metrics.

The proposed method consistently outperforms all baselines on both real-world and synthetic compositional scene generation benchmarks, achieving the best scores across all reported metrics. The largest improvements are observed on the synthetic Toys4k-Scene benchmark, where the method reduces Chamfer distance by more than an order of magnitude compared to the strongest baseline. The proposed method achieves the lowest Chamfer distance and EMD, and the highest F-Score on both HouseCat6D and Toys4k-Scene. On Toys4k-Scene, the method's Chamfer distance is over 10 times lower than the best baseline, indicating substantially more accurate scene geometry. The method's F-Score on HouseCat6D is near perfect, while the closest baseline remains below 0.98, showing superior reconstruction fidelity.

On the UE-MeshyScene dataset, the proposed method consistently outperforms the ShapeR baseline across all reported metrics, with larger gains in median than mean values, indicating robust improvement across typical objects rather than a few difficult cases. The method achieves better geometric detail and world-frame placement, especially for small and heavily occluded objects. The method surpasses ShapeR on every metric, including Chamfer distance, EMD, and F-Score. Median improvements are more pronounced than mean improvements, showing the gains are consistent across typical objects. The approach recovers more detailed geometry and more accurate placement in the shared world frame, particularly for small and heavily occluded objects.

The learned IBR aggregator generally matches or outperforms arithmetic mean fusion across most multi-view settings, with the gap widening as the number of input views and occlusion levels increase. With a single view the two methods perform nearly identically, while at higher view counts and occlusion the learned weighting yields consistent improvements in distance and fidelity metrics. The learned aggregator shows clear gains over mean fusion at high occlusion, such as reducing CD-l2 from 12.91 to 10.03 with 2 views and 75% occlusion. At 16 input views, the learned fusion improves nearly every metric across all occlusion levels compared to mean averaging. With a single input view, the two fusion strategies perform similarly, as there are no additional observations to reweight. Mean averaging only edges ahead in a few isolated settings, mostly at 8 views, and the differences are small.

The evaluation introduces UE-MeshyScene, a large-scale synthetic benchmark with diverse scene complexity and occlusion, and validates the proposed model across multiple experiments. The model remains competitive with a single view and improves consistently with more views, especially under heavy occlusion, where learned multi-view fusion clearly outperforms simple mean averaging. On both real-world and synthetic compositional scene benchmarks, the method achieves the best scores across all metrics, with particularly large gains on Toys4k-Scene where Chamfer distance drops by over an order of magnitude. The learned IBR aggregator also shows robust advantages at higher view counts and occlusion levels, while the method surpasses the ShapeR baseline on UE-MeshyScene with more pronounced median improvements, indicating consistent gains for typical objects.