One-sentence Summary

Tencent Hunyuan proposes Hunyuan3D-Buffalo 1.0, a unified framework that couples Hunyuan3D-VLM for semantic, structural, and spatial understanding with a diffusion transformer (Hunyuan3D DiT) for high-fidelity 3D synthesis to enable 3D understanding, text-to-3D generation, instruction-guided editing, and text-grounded part generation, trained on an 87M-sample multimodal corpus (25M understanding, 50M text-to-3D, and 12M editing examples), achieving state-of-the-art performance on text-to-3D and editing benchmarks alongside strong understanding and part-generation capabilities.

Key Contributions

• Proposes Hunyuan3D-Buffalo 1.0, a unified framework integrating 3D understanding, text-to-3D generation, instruction-guided 3D editing, and text-grounded part generation through a combination of Hunyuan3D-VLM and Hunyuan3D DiT.
• Constructs an 87M-scale 3D multimodal training corpus containing 25M understanding samples, 50M text-to-3D pairs, and 12M editing pairs generated via Nano3D-v2 to support scalable unified training.
• Achieves state-of-the-art or leading performance on text-to-3D generation and 3D editing benchmarks, and demonstrates that joint training of understanding and generation improves editing capability.

Introduction

The authors address the fragmentation of 3D understanding, generation, and editing systems, which stems from scarce large-scale and geometrically consistent 3D editing data and the resulting inability to learn a unified representation. To overcome this, they construct a 87M-sample 3D multimodal corpus using an agent-based editing data pipeline and propose Hunyuan3D-Buffalo 1.0, a single architecture that combines an autoregressive vision-language model with a diffusion-based 3D generator. This unified framework jointly handles 3D understanding, text-to-3D synthesis, instruction-guided editing, and text-grounded part generation, achieving state-of-the-art performance and revealing that stronger generation and understanding both directly improve editing.

Dataset

The authors build a 3D data engine that produces three complementary corpora around 3D understanding, text-to-3D generation, and 3D editing. In total the engine yields roughly 25 million understanding samples, 50 million text-to-3D pairs, and 12 million editing pairs, all used to train a unified multimodal model that can understand, generate, and edit 3D assets.

3D Understanding Data (~25M samples)

• Composition and sources: A mix of pure text, image–text, and 3D point cloud–text conversations. The text-only split (≈7M) and image–text split (≈3M) are drawn from public and in-house instruction corpora; the 3D split (≈15M) is built from Part-X-MLLM, ShapeLLM-Omni, and in-house asset pool data.
• Tasks covered:
  • 3D captioning (multi-tier descriptions of geometry, structure, and parts)
  • 3D question answering (free-form questions about category, attributes, parts, spatial relations)
  • 3D grounding (localizing parts with quantized axis-aligned bounding boxes, output as <boxs>/<boxe> token sequences)
  • Edit-instruction synthesis (producing a precise, executable editing instruction grounded in the object's geometry)
  • Edit-outcome captioning (describing the resulting object after an edit)
• Part generation data (included in the understanding corpus): Assets from HY3D-Bench and PartNeXt are processed through a semantic mesh merging tool that groups over-segmented raw mesh components into language-referable macro parts. A VLM verifies part coherence, and a single normalization transform is applied to export triplets (full object, macro part alone, remaining object). These are converted into instruction-tuning samples for part segmentation and part removal (e.g., “segment the wheels”).
• Usage: All modalities are interleaved during training to preserve language reasoning and 2D perception while adapting to 3D tasks. The understanding data is used for instruction tuning of the 3D multimodal model, covering captioning, QA, grounding, edit-instruction synthesis, and part-aware tasks.

Text-to-3D Generation Data (~50M pairs)

• Pipeline stages: A fully automated five-stage process:
  1. A four-level hierarchical prompt taxonomy (generation mode, 20 categories, subtype, fine-grained type) with controlled attribute vocabularies (style, color, material, condition, pose). Unsuitable image-to-3D styles are removed.
  2. Compositional prompt synthesis: natural user-style requests with secondary objects for hybrid composites, attribute omission (30% drop probability), and quality suffixes. Each record also carries a negative prompt.
  3. Image-to-3D asset generation and multi-view rendering (canonical, white-background views); surface point clouds are sampled. For edit-augmented splits, a difference-detection module with consistency checks discards samples with implausible changes.
  4. Multi-tier captioning and geometry-quality scoring: A VLM (Gemini) emits six caption tiers describing only geometry (no color, material, or rendering mentions) and an integer geometry-quality score (0–10, penalizing truncation, duplicated parts, broken topology, etc.).
  5. Quality filtering: Only assets with geometry-quality ≥ τ (τ=10 for the cleanest split) are kept. Surviving assets are packaged as tuples of (surface point-cloud path, caption dictionary, L1 category).
• Schema: Each sample includes {"L0", "L1", "L2", "L3", "attributes", "prompt", "negative_prompt", "secondary"?} plus multi-tier captions and the category label.
• Usage: The corpus provides large-scale text–asset pairs directly for text-to-3D generation training.

3D Editing Data (~12M pairs)

• Pipeline: Nan3D-v2, with five stages:
  1. Anchor view selection: A VLM picks the most salient view for the edit, and Qwen-Image performs 2D instruction-guided editing.
  2. Editing planning: A learned autoregressive model predicts a precise 3D bounding box from the 2D edit mask and source voxel representation.
  3. Voxel editing: A voxel Transformer performs localized FlowEdit inside the predicted box; voxels outside are replaced with originals to enforce local editing.
  4. Fine-grained refinement: LATTICE refines geometry at sub-voxel resolution; NaTex handles texture inpainting with alpha blending along a 7-voxel boundary.
  5. Annotation and filtering: A VLM assesses structural integrity, re-derives a geometry-only editing instruction (addition/replacement/removal), and verifies alignment, consistency of non-edited regions, and edit visibility. Failing pairs are discarded.
• Output: (source asset, edited asset, instruction) triplets, with both a concise and a detailed instruction focused solely on geometric changes (color, material, texture are ignored).
• Usage: The editing pairs are used to train the model for instruction-driven 3D editing while preserving non-target regions.

Method

To support unified 3D multimodal modeling, the authors construct a comprehensive data engine. For text-to-3D data, they employ a fully automated five-stage pipeline. This process begins with hierarchical prompt taxonomy construction and compositional prompt synthesis, followed by image-to-3D asset generation and multi-view rendering. The pipeline then applies multi-tier captioning with geometry-quality scoring and concludes with quality filtering.

As shown in the figure below:

For 3D editing data, the authors introduce Nano3D-v2, a framework designed to generate high-fidelity editing pairs while preserving geometric consistency. The pipeline consists of five stages: anchor view selection using a vision-language model, editing planning via a causal transformer to predict 3D bounding boxes, voxel-level editing using a voxel Transformer with FlowEdit sampling, detailed geometry and texture editing using LATTICE and NaTex Transformers, and finally, edit pairs annotation and filtering.

As shown in the figure below:

The resulting training corpus contains diverse and high-quality editing pairs, including single-round and multi-round editing scenarios.

As shown in the figure below:

As shown in the figure below:

The proposed architecture synergizes two specialized modules: Hunyuan3D-VLM for multimodal understanding and part-level reasoning, and 3D-DiT for 3D synthesis. To endow the VLM with fine-grained 3D perception, Hunyuan3D-VLM encodes 3D assets through a structure-and-appearance representation. The structural pathway processes geometric signals like XYZ coordinates and surface normals, while the semantic pathway encodes RGB appearance cues. These representations are encoded into latent tokens via a VecSet encoder and compressed by a Q-Former into a fixed-length sequence of 512 tokens. The VLM vocabulary is augmented with 133 special tokens to support explicit 3D grounding, including delimiters for point-cloud token sequences and quantized coordinate tokens for 3D bounding boxes.

To integrate the VLM with the generative module, a lightweight MLP-Connector aligns VLM hidden states with the 3D-DiT conditional space. For 3D editing and part generation, the diffusion process is conditioned on both VLM-derived semantic embeddings and the original object representation. The source 3D representation is concatenated with the noisy latent map as input to the 3D-DiT self-attention layers, facilitating the preservation of unedited regions.

The framework is trained in four distinct stages, optimizing the 3D-DiT with a flow-matching objective.

As shown in the figure below:

In Stage 1, the authors perform 3D-VLM training. This involves an alignment phase where the 3D-token connector is trained to align 3D latent tokens with the language model input space, followed by an instruction-tuning phase where the full model is jointly trained on various 3D understanding tasks. The resulting VLM is kept frozen in subsequent stages.

In Stage 2, text-to-3D pretraining couples the VLM with the 3D-DiT. VLM hidden states are processed by the MLP-Connector and injected into the 3D-DiT to condition the denoiser through cross-attention. The 3D-DiT is trained on approximately 50 million text-asset pairs.

In Stage 3, omni pretraining unifies text-to-3D, 3D editing, and part generation tasks. The sampling ratio is balanced to allow editing capabilities to emerge without sacrificing generation quality. Part generation is treated as a special case of 3D editing where the target region is the queried part.

In Stage 4, continued pre-training decouples the three tasks into task-specific paths. For 3D editing and part generation, half of the text-to-3D data is mixed in to preserve generative quality, while text-to-3D training uses exclusively text-to-3D data.

Experiment

The evaluation spans 3D understanding on UniPart-Bench, text-to-3D generation via human preference study, 3D editing on Edit3D-Bench, and open-vocabulary part generation. Hunyuan3D-VLM consistently achieves top results, unifying part-level reasoning, high-fidelity shape generation with strong text alignment, and localized edits that preserve source geometry. Scaling pretraining data is a key driver of quality, and the unified multimodal framework effectively leverages a strong text-to-3D foundation to enable accurate editing and part extraction.

A unified 3D multimodal framework is trained on a large data engine providing roughly 25M understanding dialogues, 50M text-to-3D pairs, and 12M editing triplets. The model produces localized, instruction-guided geometric edits that faithfully preserve the source shape, substantially outperforming baselines. Scaling the text-to-3D generation data emerges as a key and cost-effective factor that drives editing performance, and the same model supports open-vocabulary part generation as an integrated sub-task. The text-to-3D corpus is roughly four times larger than the editing corpus, and increasing its size improves editing capability without adding extra editing examples. Qualitative editing evaluations show localized changes (e.g., adding a sword, removing wings) that keep the overall structure intact, while baseline methods often oversmooth or distort the source shape. The framework performs open-vocabulary, text-grounded part generation with high geometric fidelity and enables compositional reassembly, all within the unified multimodal large language model.

The dataset provides six caption tiers per 3D asset, ranging from detailed multi-sentence descriptions to compact keyword lists. This multi-granular text conditioning enables precise, localized geometric edits that preserve overall shape and fine details, while also supporting open-vocabulary part extraction. Scaling the underlying text-to-3D generation data simultaneously strengthens editing performance without needing extra editing annotations. Detailed captions use 4–6 sentences (≤120 words) covering subject, parts, pose, and features, while Tags reduce to ≤8 disentangled keywords. Multi-tier captions allow localized edits such as adding accessories or removing components while maintaining the original structure and fine geometry. Scaling text-to-3D generation data improves 3D editing capability, even without additional editing-specific data. The model extracts open-vocabulary, text-specified parts with high geometric fidelity and can reassemble them into compositional outputs.

Hunyuan3D-VLM achieves the highest performance on both part-level question answering and object-level captioning among all evaluated 3D multimodal models. It delivers substantially stronger semantic similarity and lexical accuracy for fine-grained part QA, and similarly large gains for holistic object description, indicating robust part-aware and object-level 3D understanding. Part QA semantic similarity (SBERT and SimCSE) jumps to 85.47 and 89.06, marking a clear leap over the next best model. Lexical quality of part answers improves strongly, with BLEU-1 rising from 40.54 to 49.95 and METEOR from 34.24 to 45.79. Object captioning SBERT increases to 72.94 and ROUGE-L to 52.84, roughly 19 and 14 points above the strongest baseline. The model unifies fine-grained part reasoning and global semantic description, consistently outperforming prior 3D multimodal systems.

Hunyuan3D-VLM delivers consistent performance across diverse 3D understanding tasks, achieving precise pure localization and strong multi-part grounding while producing semantically accurate descriptions. The model shows a clear trade-off: fine-grained part descriptions improve semantic similarity in box-to-text generation but make single-part localization more challenging. Pure box listing attains a high IoU of 0.864, demonstrating accurate part localization without any text generation. Multi-part grounding remains robust for both coarse (Q1) and fine-grained (Q2) part labels, with IoU values of 0.880 and 0.844 respectively. Single-part grounding is more demanding: IoU drops from 0.626 for coarse part names (Q1) to 0.525 for fine-grained descriptions (Q2), and no language metrics are reported for the Q2 setting. In box-to-text generation, fine-grained Q2 descriptions yield higher semantic similarity (SBERT 74.13, SimCSE 72.99) than coarse Q1 labels, while lexical overlap metrics such as BLEU-1 and ROUGE-L are lower, suggesting meaning is captured even when exact wording differs. The model unifies part localization, region-conditioned description, and part-aware reasoning within a single framework, as shown by competitive scores across grounding, generation, and question answering tasks.

In a human evaluation using 100 diverse text prompts, the proposed model was chosen as best in a clear majority of four-way comparisons, outperforming all baselines by a wide margin across text alignment, geometry quality, and overall preference. The baselines rarely exceeded 21% preference on any criterion, well below the 25% random-choice level, confirming the method's dominant generation quality. The proposed model achieved an overall preference rate of 56.6%, roughly three times higher than the next best baseline (18.4%). For geometry quality, it was preferred 57.1% of the time, more than double the strongest competitor (21.0%). All baselines fell below 21% on every metric, indicating they were seldom selected as the best among the four candidates.

A unified 3D multimodal framework trained on a large-scale data engine produces instruction-guided, localized geometric edits that faithfully preserve the source shape and outperform baselines, while scaling text-to-3D generation data emerges as a key, cost-effective driver of editing performance without extra annotations. Multi-granular captioning supports precise part-level edits and open-vocabulary part extraction, and the same model achieves leading performance on part-level question answering, object captioning, and diverse grounding tasks, unifying localization, description, and reasoning. Human evaluations confirm the approach is strongly preferred over baselines in text alignment, geometry quality, and overall preference.