One-sentence Summary

Researchers from AI Thrust, HKUST(GZ) and TEG AIPD, Tencent propose VIBEWORLDING, a unified framework for benchmarking and training multimodal agents that infer user intent, plan scene layouts, invoke 3D tools, and reflect on multimodal feedback during end-to-end 3D open world construction; it introduces VWE-BENCH and VIBEWORLDING-GYM with rubric-based verification and agentic RL post-training, and it enables VibeWorlder-30B-A3B to surpass frontier MLLMs.

Key Contributions

• The paper presents VIBEWORLDING, a unified framework for benchmarking and training multimodal vibe worlding agents, and introduces VWE-BENCH with 2,616 3D assets, 323 human-annotated seed worlds, and 6,828 reverse-synthesized multimodal user queries split into verified and unverified sets.
• The work develops VIBEWORLDING-GYM, a joint multimodal RL post-training framework that includes a sandbox environment unifying asset retrieval, editing, and image rendering as MCP tools, along with a rubric-based verifier combining physical feasibility checks such as collision detection with intent fulfillment verification for evaluation and reward generation.
• Experiments show frontier MLLMs such as GPT-5.5 and Qwen3.8-Max remain below 60% success rate, with precise 3D world editing as the main bottleneck, while RL post-training yields VibeWorlder-8B and VibeWorlder-30B-A3B, the latter achieving the best overall Pass@1 among evaluated models with data, code, and models released.

Introduction

Constructing interactive 3D open worlds from predefined assets is central to gaming, simulation, and embodied AI, and multimodal large language models (MLLMs) have made it possible to automate this process through agentic workflows. Prior systems, however, mostly target idealized queries, remain closed-source, and operate over fragmented asset libraries and incompatible tools, while also lacking reliable verification of physical feasibility, user intent, and aesthetic coherence. The authors present VIBEWORLDING, a unified open-source framework for benchmarking and training end-to-end 3D world construction agents. It includes VWE-BENCH, with 2,616 high-quality 3D assets, 323 human-annotated seed worlds, and 6,828 queries, plus VIBEWORLDING-GYM, a multimodal RL sandbox that unifies retrieval, editing, and rendering tools and uses a dual-constraint verifier to guide evaluation and training.

Dataset

The authors construct VWE-BENCH through a three-stage collaboration between multimodal large language models (MLLMs), generative models, and human art annotators.

1. Dataset composition and sources

• 3D assets: built from an artist-defined inventory of 3,148 native concept assets with names and categories. For each asset, Gemini 3.1-flash-image generates an image, then Hunyuan3D 3.1 converts it to a 3D mesh in .glb format.
• After quality filtering, 2,616 assets remain. These span 20 semantic categories, from small props to buildings and terrain. Each asset is annotated at a unified real-world scale with center and half-extent along x/y/z axes, plus semantic description, color, native size, and face count.
• Seed worlds: professional art-annotators build 323 seed 3D worlds from the synthesized assets, jointly covering the full asset library. Worlds are coarse but functional, with 8 to 258 placed assets.
• Queries: reverse synthesis yields 6,828 multimodal queries, consisting of 1,364 3D world construction queries and 5,464 3D world refinement queries.

2. Key subsets and synthesis rules

• 3D world construction queries:
  • Theme only: high-level atmosphere only.
  • Theme + elements: theme plus required elements and quantities.
  • Full blueprint: theme, elements, and intended spatial organization.
  • Distractor: includes infeasible or under-specified subrequests, such as missing assets, physics violations, or contradictions.
• 3D world refinement queries:
  • Asset perturbation:
    • Asset-level edit precise: exact add/delete/translate/rotate instructions with a ground-truth map.
    • Asset-level edit fuzzy: vague asset-level edits, such as "tidy up the trees a bit."
  • MLLM-as-critic:
    • Scene critique: asks the agent to fix a shortcoming.
    • Scene guidance: gives high-level directional intent.
    • Scene restatement: restates the desired end state without concrete edits.
    • Complex description: multiple coordinated edits or rich requirements.
• Only asset-level edit precise queries are verified. The other five query types are unverified and evaluated with rubrics.

3. Processing details and metadata

• No cropping strategy is described. Processing includes image-to-3D conversion, filtering of meshes that differ significantly from concept images, real-world size annotation, and native bounding box annotation.
• Seed worlds are built as coarse but functional scenes, not highly polished environments.
• Query construction uses MLLMs to read seed worlds or observe perturbations, then produce instructions. Human annotators review and filter low-quality synthesized queries.

4. Data usage in the model

• The dataset is split into training and testing sets with completely disjoint seed 3D worlds to measure generalization instead of memorization.
• The split preserves query type proportions, though exact train/test counts and mixture ratios are not reported.
• Training uses the SFT query set for cold-start data synthesis and the RL query set for joint multimodal reinforcement learning.

Method

The authors propose VIBEWORLDING-GYM to support scalable agentic reinforcement learning (RL) training for 3D world construction. Given a multimodal query $q$q, the agent interacts with the sandbox environment for $T$T turns to construct an interactive 3D world. At the $i$i-th turn, conditioned on the query and interaction history, the agent generates a thought $\tau_i$τi​ and an action $a_i$ai​ consisting of tool calls:

where $o_i$oi​ is the observation returned by the sandbox, containing tool responses and the current 3D map with rendered multi-view images. The framework comprises three core components: a stable 3D sandbox for agent interaction, a dual-constraint verifier for end-to-end evaluation, and a unified post-training pipeline.

3D Sandbox Construction The sandbox provides the agent with a set of unified 3D tools and a rendering service. The toolset includes asset retrieval, addition, deletion, rotation, and translation. The retrieval tool maps natural-language intents to placeable asset candidates using a retriever built upon Qwen3-Embedding-4B, trained with InfoNCE loss on synthetic positive-negative pairs. The remaining tools manipulate the 3D world based on provided parameters. To support these operations, the authors construct a large, physically consistent asset library through an artist-guided synthesis pipeline. As shown in the figure below, this library spans diverse semantic categories and physical sizes, ranging from small props to large buildings, and serves as the foundation for constructing seed 3D worlds of varying complexity.

The statistical distributions of the synthesized assets across semantic categories and physical size classes, along with the complexity of the seed worlds, are illustrated in the following figure.

For rendering, the sandbox uses Blender to generate multi-view images of the modified 3D world from five fixed camera viewpoints after each agent turn, providing visual feedback for subsequent actions.

Dual-Constraint Verifier To evaluate the constructed 3D worlds, the authors design a dual-constraint verifier that checks both physical feasibility and intent fulfillment. Physical feasibility is verified through Python-based geometric checks ensuring assets do not collide and are properly grounded without floating. Intent fulfillment is assessed by an MLLM-based judge evaluating ecological plausibility, 3D understanding, 3D reasoning, and retrieval plausibility. For unverified queries, a world must pass both checks to be considered correct. For verified queries with ground-truth maps, the verifier scores the world based on the proportion of correctly modified assets.

Joint Multimodal RL Post-Training The training process begins with cold-start data synthesis for supervised fine-tuning (SFT). For unverified queries, the authors prompt an MLLM to construct high-quality 3D worlds, filter them using the verifier, and reverse-prompt the MLLM to generate coherent reasoning trajectories. For verified queries, trajectories are reverse-synthesized directly from ground-truth maps. Full-parameter SFT endows the model with basic 3D tool use and multi-turn reasoning capabilities.

Following SFT, the authors employ Joint Multimodal RL using Group Relative Policy Optimization (GRPO). The agent learns jointly from pure-text queries (constructing worlds from scratch) and multimodal queries (refining existing worlds given renders). The optimization relies on an outcome-based reward derived from the dual-constraint verifier: a binary reward for unverified queries and a proportional score for verified queries. This design avoids reward hacking associated with hand-crafted intermediate shaping.

Experiment

VWE-BENCH evaluates 3D world construction and refinement using disjoint training and test worlds, with automatic verifier scoring cross-checked against blind human labels. The experiments compare frontier MLLMs, agent-scaffold baselines, and post-trained VibeWorlder variants, showing that cold-start SFT plus multimodal RL lifts open backbones from low Pass@1 to state-of-the-art, with VibeWorlder-30B-A3B surpassing GPT-5.5 and Qwen3.8-Max, especially on rule-checkable verified edits. Analysis indicates SFT mainly provides physical and ecological competence, while RL unlocks 3D understanding and reasoning, though collision-free placement, precise distance editing, and occasional overediting remain key limitations. A real-world CLI study further confirms that the trained agent supports interactive multi-turn edits through render-in-the-loop verification.

The benchmark defines 3D assets, 3D worlds, multimodal queries, and a set of unified 3D tools. It includes thousands of assets, hundreds of worlds, and thousands of multimodal queries spanning from-scratch construction and refinement of existing scenes. The tool set supports asset retrieval plus add, delete, rotate, and translate operations for scene editing. The asset library contains thousands of 3D assets, each described by id, name, category, face count, and bounding box. Multimodal queries cover both building a world from a text instruction and refining an existing world using textual and visual inputs. Five unified tools support retrieval and manipulation of placeable assets, including add, delete, rotate, and translate operations.

VWE-BENCH contains over five thousand verified queries split between 3D world construction and 3D world refinement. Refinement is the larger task family, with asset-level edits contributing the most queries and scene critique contributing the fewest. Construction queries span increasing specificity tiers plus distractors, with theme plus elements being the most common subtype. Refinement queries outnumber construction queries by more than two to one, driven mainly by asset-level edit tasks. Within construction, theme plus elements is the largest subtype and distractor is the smallest; within refinement, precise asset edits lead and scene critique is least frequent.

VWE-Bench splits its dataset into training and test sets built from disjoint seed 3D worlds, supporting generalization rather than memorization. Refinement tasks are substantially more common than construction tasks across all splits, and the training data are divided into a larger SFT cold-start set and a smaller RL set. The split also preserves query type proportions across training and testing. 3D world refinement examples outnumber 3D world construction examples by roughly four to one in every split. The SFT cold-start training set is the largest subset, with fewer RL training examples and an even smaller test set. Training and test examples use completely disjoint seed 3D worlds, preventing leakage across scenes. Query type proportions are preserved between training and testing to support balanced policy learning.

Pass@1 varies widely by query subtype across evaluated models. Construction from scratch and long or underspecified refinement queries remain difficult, while precise and fuzzy asset edits and scene critique are relatively stronger for several frontier models. Multimodal RL post-training substantially improves open backbones and yields the best overall pass rate, though human judgment stays stricter. Construction subtype performance is uneven: some frontier models score highly on Theme and Full-blueprint or Distractor prompts, while others score very low or zero on the same categories. Refinement queries separate models more clearly, with Complex Description and Scene Restatement remaining difficult even for strong models, which score below 50% on those subtypes. RL post-training produces consistent improvements across refinement subtypes and lifts the open VibeWorlder-30B-A3B model to the highest overall pass rate, ahead of frontier models. Human holistic pass rates are consistently lower than overall pass rates, indicating that benchmark success does not fully capture human judgments of feasibility and intent.

The benchmark evaluates 3D world construction and refinement using multimodal queries along with a unified set of asset retrieval and editing tools. Dataset splits use disjoint seed worlds to test generalization rather than memorization, and refinement tasks are substantially more common than construction tasks. Evaluations show that construction from scratch and underspecified refinement remain difficult, while multimodal RL post-training consistently improves open models and yields the best overall pass rates. Human holistic judgments are stricter than automated pass rates, indicating that benchmark success does not fully capture human assessments of feasibility and intent.