One-sentence Summary

This paper presents O-Voxel, an omni-voxel representation enabling structured latent learning from native 3D data to model arbitrary topology and detailed appearance, employing a Sparse Compression VAE and 4B parameter flow-matching models to achieve efficient inference with geometry and material quality far exceeding existing models.

Key Contributions

• The paper introduces O-Voxel, a new sparse voxel structure that encodes both geometry and appearance. This omni-voxel representation robustly models arbitrary topology, including open and non-manifold surfaces, while capturing physically-based rendering parameters.
• A Sparse Compression VAE is designed based on O-Voxel to provide a high spatial compression rate and a compact latent space. This approach learns a compact structured latent space directly from native 3D inputs to achieve higher spatial downsampling rates and fewer latent tokens.
• Large-scale flow-matching models comprising 4B parameters are trained for 3D generation using diverse public 3D asset datasets. Experiments demonstrate that the geometry and material quality of the generated assets far exceed existing models while maintaining highly efficient inference.

Introduction

Recent advancements in 3D generative modeling aim to support industrial content creation, yet existing representations often fail to capture complex topologies and detailed appearance simultaneously. Prior methods relying on iso-surface fields struggle with open or enclosed structures, while structured latent approaches frequently depend on multiview 2D features that limit geometric fidelity. To address these issues, the authors introduce O-Voxel, a native sparse voxel structure that encodes both geometry and physically-based rendering parameters without lossy preprocessing. They further design a Sparse Compression VAE to achieve 16x spatial downsampling, enabling large flow-matching models to generate high-resolution assets efficiently in a compact latent space.

Dataset

• Dataset Composition and Sources: The authors curate a collection of 3D assets based on the TRELLIS pipeline. They explicitly exclude the 3D-FUTURE dataset because it lacks the Physically-Based Rendering (PBR) materials required for their work.
• Training Data Generation: To create image prompts for the image-conditioned model, diverse views are rendered for each 3D asset using Blender.
• Augmentation and Processing: Several rendering augmentations are applied to ensure the model remains robust against real-world ambiguities:
  • Field of View: The camera FoV is randomly sampled between 10° and 70° to account for unknown camera intrinsics.
  • Lighting Conditions: Light source placement and intensity are randomized to help the model disentangle intrinsic PBR attributes from environmental illumination.

Method

The proposed framework aims to generate high-resolution 3D assets with arbitrary topology and flexible material attributes. The overall pipeline, illustrated in the framework diagram, consists of three main stages: instant bidirectional conversion to a native representation called O-Voxel, compression into a compact latent space (SLat) via a Sparse Compression VAE, and large-scale generative modeling.

O-Voxel: A Native 3D Representation

The core of the method is the O-Voxel representation, which stores feature tuples associated with sparse voxels on a regular 3D grid. Each active voxel $i$i contains a shape feature $f_i^{\mathrm{shape}}$fishape​ and a material feature $f_i^{\mathrm{mat}}$fimat​.

Shape: Flexible Dual Grid To represent surfaces with arbitrary topology, the authors employ a Flexible Dual Grid formulation. In this grid, one vertex is defined per primal cell, and one quadrilateral face exists per primal edge connecting dual vertices in adjacent cells. This formulation allows the algorithm to adjust dual vertex positions and face existence to accurately represent input surfaces without watertight constraints.

The geometric feature $f_i^{\text{shape}}$fishape​ comprises dual vertex positions $v_i$vi​, edge intersection flags $\delta_i$δi​, and splitting weights $\gamma_i$γi​. The dual vertex position is determined by minimizing a Quadratic Error Function (QEF) that includes terms for plane distance, boundary edge distance, and regularization. This design ensures sharp feature preservation and robust handling of open surfaces.

Material: Volumetric Surface Attributes Material attributes are modeled using physically-based rendering (PBR) parameters stored volumetrically. For each active voxel, the material feature $f_i^{\mathrm{mat}}$fimat​ includes base color, metallic ratio, roughness, and opacity. These attributes are sampled from texture maps during conversion and reconstructed via trilinear interpolation during decoding, ensuring alignment with the surface geometry.

Sparse Compression VAE (SC-VAE)

To facilitate efficient generation, the authors design a Sparse Compression VAE to learn a compact latent space from O-Voxel data. The network architecture follows a U-shaped design with sparse convolutions, as illustrated in the network structure diagram.

The SC-VAE employs a Sparse Residual Autoencoding mechanism to handle high spatial compression efficiently. During downsampling, the encoder aggregates the eight children of a voxel into the channel dimension of the parent voxel. Conversely, the decoder distributes coarse features back to the neighborhood during upsampling. This non-parametric residual shortcut mitigates optimization challenges and preserves spatial information. The network utilizes ConvNeXt-style residual blocks and an early-pruning upsampler to further enhance efficiency and reconstruction quality.

The SC-VAE is trained in two stages. Initially, low-resolution data is used with direct reconstruction and KL losses to stabilize learning. Subsequently, rendering-based perceptual supervision is added at high resolution to enhance fidelity. To facilitate sequential generation, decoupled latent spaces are learned: one for shape and another for material conditioned on the shape's subdivision structures.

Generative Modeling

Built upon the learned latent space, the authors construct a scalable generative framework using Diffusion Transformers (DiT) trained with flow matching. The generation process unfolds in three stages: sparse structure generation, geometry generation, and material generation. This sequential approach allows for the synthesis of shapes at resolutions exceeding the training scale through cascaded application, leveraging the high spatial compression achieved by the SC-VAE.

Experiment

The evaluation setup employs curated benchmarks and disjoint image prompts to assess reconstruction, generation, and texture synthesis capabilities against state-of-the-art baselines. Results demonstrate that the proposed framework consistently achieves superior geometric fidelity and material realism, effectively preserving fine details, complex internal structures, and physically accurate PBR textures. Qualitative comparisons and user studies further confirm enhanced visual consistency and prompt alignment, while ablation analyses verify the critical role of the sparse residual architecture in maintaining quality under compression.

The authors compare their image-to-3D generation approach against multiple state-of-the-art baselines using automated alignment metrics and human preference studies. Their method consistently achieves the highest scores across all alignment evaluations, indicating superior visual and geometric consistency with input images. Furthermore, human evaluators demonstrate a strong preference for their generated assets over competitors in both overall quality and shape accuracy. The proposed method achieves the top performance in all automated alignment metrics compared to baselines like TRELLIS and Hunyuan3D 2.1. Human preference studies reveal a significant advantage for the proposed method in both overall visual quality and geometric shape fidelity. Several baseline methods show gaps in performance, with some lacking specific alignment metrics or receiving notably lower user preference ratings.

The authors conducted a user study to evaluate the perceptual quality of their 3D generation framework against several state-of-the-art baselines. The results demonstrate that the proposed method was overwhelmingly preferred by participants for both overall visual quality and shape accuracy. This indicates a clear superiority in generating assets with high visual fidelity and precise geometric details compared to existing approaches. The proposed method achieved the highest preference rate among all tested baselines in the user study. Participants favored the authors' approach for both overall visual quality and shape fidelity over competitors like Hunyuan3D 2.1 and TRELLIS. The results indicate a substantial performance gap, with the proposed method receiving significantly more votes than the next best alternatives.

The authors perform an ablation study to assess the impact of the sparse residual autoencoding layer and the optimized residual block on the SC-VAE architecture. The results demonstrate that both components are essential for high-fidelity reconstruction, with the removal of the residual autoencoding layer causing the most significant degradation in geometric and appearance metrics. The full model consistently outperforms the ablated variants across different compression settings while maintaining comparable decoding efficiency. Removing the sparse residual autoencoding layer leads to a severe drop in reconstruction quality, particularly increasing geometric error and reducing texture fidelity. The optimized residual block contributes to better preservation of fine details, as its absence results in lower scores for surface quality and geometric accuracy. The complete architecture achieves the best overall performance, balancing high reconstruction fidelity with efficient decoding times across varying model configurations.

The the the table outlines the composition of datasets used for training the SC-VAE and the broader generative model, alongside a separate evaluation benchmark. It distinguishes between a curated SC-VAE training set derived from ObjaverseXL, ABO, and HSSD, and a larger comprehensive training set that integrates TexVerse to significantly increase data volume. The SC-VAE training set is exclusively composed of assets from ObjaverseXL, ABO, and HSSD. The full training set is substantially larger, achieved by augmenting the SC-VAE set with TexVerse data to enrich material diversity. Toys4k is designated as a distinct evaluation set, ensuring a clear separation between training and testing data.

The authors compare their reconstruction method against baselines such as Dora, TRELLIS, and SparseFlex using the Toys4K and Sketchfab Featured benchmarks. Their approach consistently outperforms these methods in geometric fidelity and surface quality metrics while utilizing fewer tokens and achieving faster decoding speeds. The method achieves superior geometric accuracy and surface detail preservation compared to all baselines. High reconstruction quality is maintained across both simple and complex asset datasets. The approach demonstrates significantly faster decoding times and lower token usage than competing high-fidelity methods.

The authors validate their image-to-3D framework using curated training datasets alongside automated alignment metrics, human preference studies, and ablation analyses against leading baselines. Their method consistently demonstrates superior visual quality and geometric fidelity in generation tasks, while ablation results confirm that specific architectural components are critical for high-fidelity reconstruction. Furthermore, the proposed approach outperforms competing methods in reconstruction benchmarks by maintaining detailed surface quality with greater decoding efficiency.