One-sentence Summary

Meta Superintelligence Labs introduces SAM 3D, a generative model that reconstructs object geometry, texture, and layout from a single image, excelling in natural scenes with occlusion and clutter via a human- and model-in-the-loop annotation pipeline paired with a multi-stage synthetic-to-real training framework, achieving at least a $5:1$5:1 human preference win rate and releasing code, model weights, an online demo, and a benchmark.

Key Contributions

• Introduces SAM 3D, a foundation model that predicts 3D shape, texture, and pose from a single natural image, addressing occlusion and clutter in in-the-wild scenes where prior single-view methods handle poorly.

• Builds a model-in-the-loop and human-in-the-loop annotation pipeline, where annotators select or refine 3D shape proposals from computational and retrieval-based modules, and the vetted annotations feed back into model training, creating a data engine that produces visually grounded 3D reconstruction data at unprecedented scale.

• Adopts a multi-stage training framework combining synthetic pretraining with real-world post-training alignment, achieving a 5:1 win rate over prior state of the art on real-world objects and 6:1 on scenes in human preference tests, and releases the SA-3DAO benchmark of 1,000 artist-created 3D meshes paired with natural images, along with code, model weights, and an online demo.

Introduction

Single image 3D reconstruction has traditionally relied on multi-view geometry, but humans can infer shape from pictorial cues such as shading, texture, and object recognition. Recent generative methods achieve strong results on isolated synthetic objects, yet they struggle in natural scenes where objects may be distant or heavily occluded. A core bottleneck is data scarcity: natural images paired with 3D ground truth are difficult to obtain at scale, and generalist annotators cannot easily produce 3D shape models for arbitrary objects. The authors address this with SAM 3D, a generative neural network that reconstructs full 3D shape, texture, and camera layout from a single image. They introduce a multistage training recipe inspired by LLMs, starting with synthetic pretraining, followed by semi-synthetic mid-training, and finishing with post-training on real images using a model-in-the-loop pipeline where annotators select and align 3D proposals. They also release SA-3DAO, a benchmark of 1,000 artist-created 3D meshes paired with natural images, to evaluate real-world reconstruction. Their approach achieves a 5:1 win rate over prior state of the art on real-world objects and 6:1 on scenes in human preference studies.

Dataset

Dataset Composition and Sources

The authors build datasets across three training stages, each targeting a different capability. All data is synthetic or semi-synthetic for pre and mid-training, while post-training introduces real-world images through a human-in-the-loop data engine.

Key Subsets

• Iso-3DO (Pretraining)

  • Source: 2.7 million object meshes from Objaverse-XL and licensed datasets.
  • Composition: image-shape-texture triplets rendered from 24 viewpoints, each yielding a high-resolution image of a single centered object.
  • Scale: trained for 2.5 trillion tokens.

• RP-3DO (Mid-Training)

  • Source: textured meshes rendered into natural images via alpha compositing (render-paste).
  • Composition: two subsets, one with occluder-occludee pairs and another with real objects replaced by synthetic objects at similar location and scale, providing physically plausible data with accurate 3D ground truth.
  • Scale: 61 million samples with 2.8 million unique meshes, trained for 2.7 trillion tokens.

• Art-3DO (Hard Example Triage)

  • Source: professional 3D artists.
  • Composition: a small set of hardest cases where no model produces a reasonable object shape; routed to artists for direct annotation.

• Post-Training Data (Real-World Alignment Loop)

  • Source: natural images sampled from diverse real-world datasets, balanced using a 3D-oriented taxonomy.
  • Composition: annotations include 3D shape, texture, orientation, 3D location, and scale. The pipeline has three stages: target object selection (image and mask), object model ranking and selection (shape and texture), and aligning objects to a 2.5D scene (pose, location, scale relative to a point cloud).
  • Scale: almost 1 million images with 3.14 million untextured meshes and 100K textured meshes.

Filtering and Quality Rules

• Annotators choose the best among 8 candidates from a model ensemble, a best-of-N search. A model first filters candidates, then humans filter further.
• Annotators rate each example r and reject those below a quality threshold omega. Rejected candidates become negative examples for preference alignment.
• At the first iteration, the authors rely on an ensemble of learned and retrieval-based models to overcome the cold start. As training progresses, the best model produces about 80% of the annotated data.

Evaluation Datasets

• SA-3DAO: a new benchmark with 1K 3D artist-created meshes from natural images.
• ISO3D (from 3D Arena): used for quantitative shape and texture evaluation.
• Aria Digital Twin (ADT): used for layout evaluation.
• Pref Set: two curated sets for human preference evaluation at scene and object levels, using real-world images from MetaCLIP, SA-1B, and LVIS.

How the Data Is Used

• Pre and mid-training use rectified conditional flow matching to generate multiple 3D modalities, building priors for shape, texture, mask-following, occlusion robustness, and layout estimation.
• The post-training data engine is framed as an API that takes a current model and returns training samples, quality ratings, and less preferred candidates for preference alignment.
• In experiments, a Geometry model is trained with pointmap conditioning for shape and texture, while layout evaluation requires ground truth depth or pointmaps as reference.

Method

The authors propose a two-stage latent flow matching architecture designed to reconstruct 3D objects from 2D images. The system operates in two primary phases: first predicting coarse geometry and layout, and subsequently refining the shape and synthesizing texture. The model is conditioned on input images and object masks. Specifically, a DINOv2 encoder extracts features from two pairs of images: the cropped object region and the full image. This provides both high-resolution local details of the target object and global scene context. Optionally, the model can integrate a coarse point map obtained from hardware sensors or monocular depth estimation.

The first stage is the Geometry Model, which predicts the conditional distribution of coarse shape $O$O and layout parameters $(R, t, s)$(R,t,s), representing 6D rotation, translation, and scale. This module employs a Mixture of Transformers (MoT) architecture. As shown in the framework diagram below, the MoT utilizes a two-stream approach where shape and layout tokens are processed in parallel streams. These streams interact through a multi-modal self-attention layer, governed by a specific attention mask that controls information flow between the different token types. The output of this stage is a coarse voxel representation of the object and its spatial layout.

The second stage is the Texture & Refinement Model, which learns the conditional distribution of high-resolution shape $S$S and texture $T$T given the coarse geometry and image conditioning. This module uses a sparse latent flow transformer that takes the active voxels predicted by the Geometry Model as input. It refines the geometric details and synthesizes the object texture. The resulting latent representations are decoded into either a mesh or 3D Gaussian splats using separate Variational Autoencoder (VAE) decoders that share a common structured latent space.

To bridge the gap between synthetic data and real-world applications, the authors employ a multi-stage training strategy that progresses from synthetic pre-training to a real-world alignment loop. The training begins with Pre-training on isolated synthetic objects (Iso-3DO dataset), teaching the model foundational shape and texture generation capabilities. This is followed by Mid-training on semi-synthetic data (RP-3DO dataset), which involves rendering textured meshes into natural images with artificial occluders. This stage instills critical skills such as mask-following, occlusion robustness, and layout estimation. The diversity of these data sources, ranging from isolated assets to artist-annotated meshes, is illustrated in the figure below.

The core of the training methodology is the Post-Training phase, which utilizes a data engine to iteratively collect real-world data and align the model with human preferences. This process involves a multi-stage collection pipeline:

1. Stage 1 (Object Selection): Human annotators identify target objects in real-world images and provide segmentation masks.
2. Stage 2 (Model Ranking): The model generates candidate 3D shapes, which are ranked and selected by annotators based on visual fidelity.
3. Stage 2.5 (Artist Triage): Hard cases that models cannot resolve are routed to professional 3D artists for direct annotation.
4. Stage 3 (Scene Alignment): Annotators align the selected 3D shapes to the 2.5D scene by adjusting rotation, translation, and scale relative to a point cloud.

As depicted in the pipeline diagram below, this human-in-the-loop process generates high-quality training pairs $(I, M) \rightarrow (S, T, R, t, s)$(I,M)→(S,T,R,t,s).

The collected data is then used for Model Improvement. This includes Supervised Fine-Tuning (SFT) on the aligned meshes, starting with noisier labels and progressing to high-quality artist annotations to suppress common failures. To further align with human aesthetic preferences, the authors apply Direct Preference Optimization (DPO) using the ranked candidate pairs generated during data collection. Finally, a distillation step reduces the number of function evaluations required during inference, enabling sub-second generation from the Geometry model.

Experiment

Experiments evaluate SAM 3D on a newly built SA-3DAO benchmark of 1K real-world 3D meshes, alongside ISO3D, ADT, and human preference sets. SAM 3D outperforms prior state-of-the-art in with a 5:1 win rate for single-object generation and 6:1 for scene reconstruction, especially on challenging real-world inputs, and introduces joint shape and layout capability. Analysis shows steady near-linear performance gains from longer data engine iterations and scaling all training stages together, with multi-stage training yielding near-monotonic improvements in shape quality.

SAM 3D employs a multi-stage training pipeline that progressively integrates geometry, texture, and refinement stages, with each stage contributing to steady performance gains. The model achieves strong human preference win rates and significantly outperforms prior methods on real-world inputs, while remaining modular to depth estimators. Multi-stage training yields near-monotonic shape improvements as each stage is added. SAM 3D achieves a 5:1 head-to-head win rate over prior SOTA on real images in human preference studies. User preferences favor SAM 3D scene reconstructions by a 6:1 margin over prior SOTA. The model exhibits significant gains on challenging real-world datasets with heavy occlusion. Performance scales with more data engine iterations, showing near-linear Elo improvements.

The comparison shows that SAM 3D matches or exceeds prior state-of-the-art on isolated object images (ISO3D) and significantly outperforms all baselines on challenging real-world inputs (SA-3DAO). Among the listed baselines, Hi3DGen achieves the highest [email protected] and vIoU on SA-3DAO, while TripoSG benefits from a higher mesh resolution in perceptual metrics. SAM 3D significantly outperforms all baselines on real-world SA-3DAO inputs. On ISO3D, SAM 3D matches or exceeds previous state-of-the-art performance. Among baselines, Hi3DGen leads on SA-3DAO accuracy metrics, while TripoSG's higher mesh resolution boosts perceptual scores.

In 3D layout comparisons on SA-3DAO and Aria Digital Twin, SAM 3D combined with FoundationPose achieves the strongest overall performance, particularly on Aria Digital Twin where it yields the highest 3D IoU and [email protected] while minimizing rotation error and ADD-S. Pipeline baselines show variable results, and the joint generative model MIDI performs poorly, with much higher errors and near-zero overlap. SAM 3D + FoundationPose outperforms all pipeline baselines on Aria Digital Twin, with the highest 3D IoU and [email protected] and the lowest rotation error. On SA-3DAO, SAM 3D + FoundationPose ranks among the best for rotation error and [email protected], though HY3D-2.0 + FoundationPose shows a slightly higher 3D IoU. The joint generative model MIDI performs drastically worse than pipeline approaches, with near-zero 3D IoU and very high rotation and ADD-S errors on Aria Digital Twin.

Multi-stage training progressively improves 3D shape quality, with each added stage yielding better F1, vIoU, and lower Chamfer and EMD scores. Texture win rates also rise with intermediate stages, though the final DPO on Art-3DO shows slight preference fluctuation. The consistent gains validate the approach of scaling stages simultaneously. Shape metrics improve nearly monotonically as each training stage is added, from pre-training through DPO on Art-3DO. The largest relative gains in shape quality occur when moving from pre-training to mid-training, with subsequent stages contributing smaller but steady improvements. Texture win rates increase with mid-training and SFT, then hold roughly steady at the DPO stage, indicating sustained preference for later-stage outputs.

The evaluation validates SAM 3D through a multi-stage training pipeline that progressively integrates geometry, texture, and refinement, yielding near-monotonic shape improvements and strong human preference win rates over prior state-of-the-art on real-world images. It outperforms all baselines on challenging real-world inputs and matches or exceeds prior performance on isolated object images, while remaining modular to depth estimators and scaling with more data engine iterations. In 3D layout tasks, SAM 3D combined with FoundationPose achieves the strongest overall performance on real-world datasets, with the joint generative model MIDI drastically underperforming compared to pipeline approaches.