One-sentence Summary

Researchers from ByteDance Seed, Peking University, and Zhejiang University propose Lucida, a composable real-to-sim scene modeling pipeline that parses video into per-instance multi-view evidence, generates complete editable assets, and places them via GizmoAct, a VLM policy using closed-loop multi-turn GUI gizmo manipulation, improving mAP over Boxer by 69% on R2S-Scene, raising [email protected] from 57.8% to 83.4% on CA-1M, and increasing scene F-Score from 0.794 for SAM 3D to 0.924.

Key Contributions

• Lucida, a composable scene modeling system, keeps the parse-generate-place order but redistributes requirements so parsing consumes multi-view instance evidence, generation completes amodal assets, and precision is reached at the final placement step.
• GizmoAct is a vision-language-model policy that casts object placement as multi-turn GUI interaction, manipulates a 3D gizmo in closed loop, decides its own alignment termination, and refines full 9-DoF pose and anisotropic scale.
• Experiments show that Lucida improves scene-level 3D object detection mAP over Boxer by 69% on R2S-Scene, raises [email protected] from 57.8% to 83.4% on CA-1M, and increases scene F-Score from 0.794 for SAM 3D to 0.924.

Introduction

Composable scene modeling recovers real indoor spaces as separable, editable object assets for robot simulation, embodied AI, AR/VR, and content creation. Prior approaches either produce faithful but monolithic reconstructions, return simulation-ready scenes limited by CAD database coverage, or synthesize plausible rooms that are not grounded in a specific capture; multi-stage parse-generate-place pipelines also assume clean masks and unoccluded views that cluttered captures do not provide, so early errors propagate. The authors introduce Lucida, a system that keeps the parse-generate-place order but redistributes precision to a closed-loop final placement stage. Parsing builds a scene graph with per-instance multi-view evidence, generation conditions on that evidence to complete occluded objects, and GizmoAct, a vision-language-model policy, aligns generated assets through multi-turn GUI interaction, tolerating mismatched geometry and coarse initial poses.

Method

The authors present Lucida, a pipeline that converts posed RGB-D observations of an indoor scene into an editable, object-level replica. The overall framework follows a parse-generate-place order, where each real instance becomes a complete 3D asset placed by a 9-DoF pose and organized in a scene graph.

In the parsing stage, the system consolidates multi-view observations into an object-centric scene graph. Given posed RGB-D observations, the authors build a graph where each node carries a per-instance evidence bundle $\mathcal{E}_o$Eo​. This bundle contains multi-view observations, associated masks, partial point-cloud observations, a representative 3D box, and a category name. To ensure robust object discovery, the method employs geometry-aware keyframe selection based on covisibility and temporal separation, followed by object-centric full-sequence evidence consolidation to retrieve additional observations from the entire sequence. Relation-aware scene refinement is then applied to correct scene-level errors such as incorrect grouping and spatial inconsistencies.

The generation stage converts each evidence bundle into a complete standalone object asset. Since real captures often suffer from occlusion and noisy depth, the authors first use multi-view visual evidence to synthesize a complete, occlusion-free object-centric image. A vision-language model organizes selected reliable views into an anchor and references to produce an editing instruction. An image-editing model then synthesizes the isolated object image, which is subsequently lifted into a 3D asset $A_o$Ao​ via an image-to-3D model.

The placement stage, termed GizmoAct, grounds the generated asset through closed-loop gizmo manipulation. The authors reformulate 3D grounding as a multi-turn GUI interaction. The state of an object is defined as $x_t = (p_t, R_t, s_t)$xt​=(pt​,Rt​,st​), representing the object center, rotation, and anisotropic scale. At each turn, the current state is rendered into a graphical observation, and a vision-language model predicts an executable edit.

The graphical observation interface pairs the scene point cloud with interface elements, including the 3D model overlay, its 3D box, and a gizmo exposing the local coordinate frame. To provide comprehensive spatial context, the system renders main views, auxiliary views for additional multi-view evidence, and orthographic views along the local axes to resolve scale-depth ambiguity. An occlusion cue is also overlaid to clarify depth relations between the model and the point cloud.

The core action space consists of update_pose for incremental edits and stop to end the episode. The update_pose action predicts rotation, translation, and scale deltas relative to the current object size and local frame, avoiding the ambiguity of absolute pose prediction. To handle large initial rotation errors that are difficult to resolve with incremental updates alone, the authors introduce extended actions for coarse rotation.

When the initial rotation error is substantial, the policy issues a switch_obs action to render the asset along six signed axes, exposing hidden sides. From these orthographic observations, the policy predicts a permute_axis action to select one of the 24 axis-aligned reorientations, removing the dominant rotation residual in a single step before fine-tuning with update_pose.

The training process for GizmoAct involves two phases. First, the authors perform supervised finetuning on synthetic expert trajectories. To teach the policy error recovery, they inject corrupted commands into the expert rollouts and supervise the subsequent recovery actions. The loss function masks out the injected error tokens. Second, they refine the policy using reinforcement learning with GRPO on online rollouts. The reward design quantizes the generalized 3D IoU and geodesic rotation error into discrete levels, providing a joint success bonus and an excellent level bonus, while dynamic sampling ensures informative batch training by rejecting groups with uniform rewards. Finally, optional postprocessing enforces scene-level constraints like support relations and collision consistency.

Experiment

The evaluation covers scene-level 3D object detection, object pose estimation and layout refinement, and full scene reconstruction, followed by ablations of the parsing pipeline and GizmoAct training. Detection experiments show that Lucida recovers object instances and 3D boxes more accurately than baseline methods, especially when using keyframe prompts with full-sequence evidence consolidation. Pose refinement evaluations demonstrate that GizmoAct improves alignment and oriented-box overlap across different datasets and initializers, with reinforcement learning enabling better recovery from large initialization errors. Scene reconstruction results indicate that the complete system produces more consistent object geometry and relative placement, and ablations confirm that geometry-aware keyframe selection, object-centric evidence consolidation, and relation-aware refinement each contribute to the overall gains.

Lucida achieves the highest scene-level 3D object detection mAP across all four evaluation settings on CA-1M and R2S-Scene. It surpasses the strongest baseline, even when that baseline uses prompts on every frame, while Lucida starts from only keyframe prompts. The largest gain appears on R2S-Scene under the all-annotations protocol, where mAP nearly doubles. On CA-1M, Lucida with keyframe prompts outperforms Boxer with all-frame prompts in both the all-annotations and filtered evaluation protocols. On R2S-Scene under the all-annotations protocol, Lucida raises mAP from 0.351 to 0.592 compared to the prior best result. Using only geometry-aware keyframe prompts, Lucida still beats Boxer when Boxer is given prompts on every annotated frame, showing the benefit of full-sequence evidence consolidation and relation-aware scene refinement.

Across R2S-Object, CA-1M, and ADT, GizmoAct improves strict surface alignment and 3D IoU over all baselines. Using up to four views further improves pose estimates relative to the single-view variant. Training the refinement policy on Boxer-initialized poses also improves accuracy over random pose perturbations. GizmoAct achieves lower ADD-SB and higher strict [email protected] success and 3D IoU than Any6D, SAM 3D, and RecGen. The max-4-view GizmoAct variant improves over the single-view variant on R2S-Object, reaching the highest reported strict alignment and 3D IoU. Matching the RL training pose distribution to Boxer initialization improves [email protected] and 3D IoU across datasets.

The same GizmoAct policy is evaluated under three pose initializers on three datasets. Boxer initialization tends to be strongest on R2S-Object and CA-1M, while SAM 3D and Any6D* initialization are more competitive on ADT's exact geometry. Initializer choice affects surface error, strict pose success, and 3D IoU even with a fixed multi-view refinement budget. On R2S-Object and CA-1M, Boxer initialization achieves the lowest ADD-SB and highest 3D IoU among the three initializers, although SAM 3D records a slightly higher strict success rate on R2S-Object. On ADT, SAM 3D initialization yields the highest strict success rate and 3D IoU, while Any6D* yields the lowest ADD-SB, showing a different initializer ranking under exact geometry.

The proposed method outperforms all baselines on scene-level and object-level metrics. It more than halves the total scene Chamfer distance relative to SAM 3D, while lifting scene F-Score from 0.794 to 0.924 and bounding box IoU from 0.396 to 0.495. Gains at the object level are modest but consistent, with lower dimensionless CD and higher F-Score. Total scene Chamfer distance drops from 0.022 (SAM 3D) to 0.010, with both directed terms lower, indicating fewer spurious surfaces and better ground-truth coverage. Scene F-Score rises substantially from 0.794 to 0.924, and bounding box IoU increases from 0.396 to 0.495, reflecting more accurate object scale and relative placement. Object-level Chamfer distance (per-object normalized) improves from 0.038 to 0.034, and object F-Score from 0.704 to 0.736. SceneGen produces much higher scene Chamfer (0.428) and markedly lower F-Score and IoU, lagging far behind the other two methods.

Ablating any of the three scene-parsing stages lowers both scene-level 3D object detection mAP and scene reconstruction F-score relative to the full pipeline. Replacing geometry-aware keyframe selection with uniform sampling causes the largest drop, indicating keyframe quality is especially important. Object-centric evidence consolidation mainly affects scene coverage, while relation-aware refinement improves detection and reconstruction alignment without much effect on total scene chamfer distance. All three ablations reduce detection mAP and scene F-score. Uniform keyframe selection produces the largest decrease, showing geometry-aware selection retains more useful viewpoints. Removing object-centric full-sequence evidence consolidation increases GT-to-pred chamfer distance, consistent with reduced scene coverage. Disabling relation-aware scene refinement lowers detection mAP and scene F-score but leaves total scene chamfer distance nearly unchanged.

The experiments evaluate a unified scene understanding pipeline across scene-level detection, object pose refinement, multi-view reconstruction, and ablation studies on CA-1M, R2S-Scene, R2S-Object, and ADT. Lucida improves scene-level 3D object detection using only keyframe prompts, while GizmoAct enhances pose alignment and benefits from multi-view refinement and Boxer-initialized training distributions. The full method outperforms baselines in scene and object reconstruction, and ablations confirm that geometry-aware keyframe selection, full-sequence evidence consolidation, and relation-aware refinement all contribute, with keyframe selection being the most critical.