One-sentence Summary

ArtLLM is a novel framework that generates high-quality articulated 3D assets by utilizing a 3D multi-modal large language model to autoregressively predict a variable number of parts and joints from point clouds, a method that outperforms state-of-the-art approaches in part layout accuracy and joint prediction on the PartNet-Mobility dataset.

Key Contributions

• The paper introduces ArtLLM, a 3D multi-modal large language model that autoregressively predicts a variable number of parts and joints by outputting a tokenized blueprint of an object's layout and kinematic relationships from an input point cloud.
• This framework integrates the predicted articulation-aware layout with a part-aware generative model to synthesize high-fidelity, novel part geometries, overcoming the geometric repetition found in retrieval-based methods.
• The method incorporates a physical constraint-based limit correction mechanism to optimize joint limits and mitigate mesh collisions, with experiments on the PartNet-Mobility dataset demonstrating superior performance in part placement, joint accuracy, and generalization to real-world objects.

Introduction

Creating interactive digital environments for gaming, robotics, and simulation requires articulated 3D assets that possess both functional part geometry and accurate kinematic structures. Existing optimization-based methods are often limited by slow per-object reconstruction and a tendency to produce only simple, single-joint objects. Meanwhile, retrieval-based approaches rely on fixed part libraries, which results in repetitive geometry and poor generalization to novel shapes. The authors leverage a 3D multi-modal large language model to bridge this gap by autoregressively predicting part layouts and joint relationships from point clouds. This framework produces a tokenized kinematic blueprint that guides a generative model to synthesize high-fidelity, novel part geometries, resulting in physically grounded and diverse articulated assets.

Dataset

• Dataset Composition and Sources: The authors constructed a large scale dataset by aggregating existing articulation datasets and procedural data. The collection includes objects from PartNet-Mobility and PhysX3D, supplemented by 12k synthetic assets generated using the Infinite-Mobility procedural method. The final curated dataset contains 20,673 articulated objects across 43 categories.

• Data Processing and Filtering:

  • Filtering: The authors removed objects with more than 20 joints and excluded categories with excessively small parts (such as keyboards or remotes). Additionally, small components like buttons were filtered out based on volume thresholds.
  • Structure Simplification: To reduce prediction complexity, all fixed joints were removed and their connected links were merged. Screw joints, typically represented by a combination of revolute and prismatic joints in URDF files, were merged into a single screw joint type.
  • Normalization: All joint parameters and geometries were transformed into the global coordinate frame and normalized to a range of $[-0.9, 0.9]$[−0.9,0.9].
  • Normal Correction: For models in PartNet-Mobility with incorrect surface normals, the authors applied watertight reconstruction to ensure accuracy.

• Training and Evaluation:

  • Training: The authors use the curated dataset for multi task training. During this process, point clouds are replaced with point tokens produced by a point cloud encoder. The training data is formatted in ShareGPT style, utilizing concise text prompts to differentiate between tasks while minimizing token usage. The model is trained on both continuous values and discretized values.
  • Evaluation: The evaluation is conducted using the PartNet-Mobility dataset. Following the SINGAPO split, the authors selected 7 categories (Storage, Table, Refrigerator, Dishwasher, Oven, Washer, and Microwave) comprising 77 held out objects for the test set. Generalization ability is further assessed using real world images, including objects from categories not present in the test set.

Method

The authors propose a framework for generating articulated assets from point clouds, which consists of three main stages: kinematic structure prediction, part geometry synthesis, and physical limit correction. As shown in the framework diagram below:

The first stage utilizes a novel 3D articulation language model (ArtLLM) to autoregressively predict the object's kinematic structure. To leverage the reasoning capabilities of Large Language Models (LLMs), the authors reformulate 3D articulation understanding as a language modeling problem. The entire kinematic structure, including part layouts and joint parameters, is represented as a unified sequence of discrete tokens. The input is a point cloud, which is processed through a Point Transformer v3 encoder. To bridge the modality gap, the encoder features are augmented with position embeddings and then projected via a two-layer MLP to align with the Qwen3 0.6B language model backbone.

To ensure numerical stability during training, the authors employ a quantization strategy for continuous geometric and kinematic parameters. For a part's axis-aligned bounding box (AABB), coordinates are quantized from a normalized range of $[-1, 1]$[−1,1] into 128 discrete bins per axis using the following formula:

$\hat { c } _ { \mathrm { m i n } } = \Big \lfloor \frac { ( c _ { \mathrm { m i n } } + 1 ) } { 2 } \times 1 2 8 \Big \rfloor , \hat { c } _ { \mathrm { m a x } } = \Big \lceil \frac { ( c _ { \mathrm { m a x } } + 1 ) } { 2 } \times 1 2 8 \Big \rceil$c^min​=⌊2(cmin​+1)​×128⌋,c^max​=⌈2(cmax​+1)​×128⌉

Joint parameters, including origins and limits, undergo similar discretization. The model is trained using a multi-task and multi-stage supervised fine-tuning (SFT) strategy, which involves part layout prediction, kinematic prediction, and end-to-end articulation prediction. This approach decouples geometric understanding from kinematic reasoning, establishing a robust foundation for the model.

In the second stage, the predicted structural blueprint serves as a condition for a part-aware generative model, specifically XPart, to synthesize high-fidelity part geometries. To prevent geometric truncation when predicted bounding boxes do not perfectly encompass the ground-truth geometry, the authors implement a bounding box expansion step. Any point in the input cloud not contained within a predicted box is assigned to its nearest box based on Euclidean distance, and the box is subsequently expanded to tightly enclose these points.

The final stage is a physically-constrained joint limit correction module designed to prevent inter-part collisions. Since the LLM predicts joint limits based on a single geometric state, the resulting assets might suffer from collisions during motion. The authors address this by articulating the child part through its predicted range and computing the collision volume against other static parts. As illustrated in the figure below, significant collisions are identified by sharp spikes in the derivative of the collision volume relative to the joint angle. A hierarchical search is then performed within the identified angular window to find the precise angle of initial contact, which is then set as the refined, collision-free joint limit.

Experiment

The authors evaluate their approach by comparing it against state-of-the-art articulation generation methods and conducting ablation studies to validate individual module contributions. Qualitative and quantitative assessments demonstrate that the proposed method superiorly recovers accurate part layouts, joint types, and kinematic hierarchies compared to baseline models that often struggle with misaligned axes or incorrect geometry. Furthermore, a real-to-sim evaluation confirms that the generated assets faithfully preserve real-world articulation properties, making them suitable for practical robotic simulation tasks.

The authors conduct ablation studies to evaluate the impact of different components on the model's performance across several metrics. The results demonstrate that the full configuration achieves superior performance in most categories, particularly in part layout and joint type accuracy. The full model achieves the highest performance in part layout IoU, joint type accuracy, and graph accuracy. Removing multi-task learning or changing the training stage leads to declines in most key metrics. Data augmentation through random scaling and rotation is shown to be essential for improving part layout IoU.

The authors compare their proposed method against several baseline approaches across various articulation and part prediction metrics. Results show that the proposed method outperforms existing models in most categories, including part layout, joint type accuracy, and hierarchical structure modeling, while also being significantly faster during inference. The proposed method achieves superior performance in part layout and joint origin prediction compared to the baselines. While some baseline methods show competitive results in specific areas like joint range, they generally lag in joint axis and pivot error metrics. The proposed approach demonstrates a substantial reduction in inference time compared to the other evaluated methods.

The authors perform ablation studies and comparative analyses to evaluate the effectiveness of individual model components and the overall performance against existing baselines. The ablation results confirm that multi-task learning and specific data augmentation techniques are critical for achieving high accuracy in part layouts and joint types. Furthermore, the proposed method demonstrates superior modeling of hierarchical structures and joint properties while offering significantly faster inference speeds than competing approaches.