Research map / 3D vision / 3D generation
ArtFormer: Controllable Generation of Diverse 3D Articulated Objects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ArtFormer, a transformer that generates 3D articulated objects as a tree of part tokens (geometry latent, joint parameters) decoded via a diffusion-based SDF shape prior, conditioned on text or images.
Problem
Generating diverse, high-quality 3D articulated objects with both part geometry and kinematic relations, without being restricted to predefined structures or retrieval from static datasets.
Approach
Represent an articulated object as a tree of tokens (bounding box, geometry latent, joint axis, limit, parent index), generate them autoregressively with a transformer using a tree position embedding and cross-attention conditioning, and decode each part's geometry with a discrete-codebook, diffusion-based SDF shape prior so parts can be generated rather than retrieved.
Key results
- Ours achieves COV 0.5213 and 1-NNA 0.5266 versus NAP-768 at 0.3723 and 0.6543, and POR 0.709e-2 versus 1.620e-2 for NAP-768.
- Against the retrieval-based CAGE, Ours-PR reaches COV 0.6400 and 1-NNA 0.3950, while CAGE leads on MMD (0.0193 vs 0.0214) and POR (0.251e-2 vs 0.556e-2).
- In a 20-participant human study, Ours obtains alignment win rate 0.459 and diversity win rate 0.67, the highest among the compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ArtFormer instead of NAP-128 | ArtFormer · PartNet-Mobility | MMD: 0.0710 -> 0.0292 | Helped |
| Use ArtFormer instead of NAP-128 | ArtFormer · PartNet-Mobility | COV: 0.3085 -> 0.5213 | Helped |
| Use ArtFormer instead of CAGE | ArtFormer · PartNet-Mobility | COV: 0.6064 -> 0.5213 | Hurt |
| Use ArtFormer instead of CAGE | ArtFormer · PartNet-Mobility | 1-NNA: 0.5319 -> 0.5266 | No clear effect |
| Add the tree position embedding (ablation) | ArtFormer · PartNet-Mobility | POR (×10−2): 1.170 -> 0.709 | Helped |
| Add the tree position embedding (ablation) | ArtFormer · PartNet-Mobility | COV: 0.5000 -> 0.5213 | Helped |
| Add the shape prior (ablation) | ArtFormer · PartNet-Mobility | COV: 0.4574 -> 0.5213 | Helped |
| Add the shape prior (ablation) | ArtFormer · PartNet-Mobility | 1-NNA: 0.7606 -> 0.5266 | Helped |
| Use one codebook instead of multiple codebooks (ablation) | ArtFormer · PartNet-Mobility | COV: 0.5213 -> 0.4948 | Hurt |
| Use one codebook instead of multiple codebooks (ablation) | ArtFormer · PartNet-Mobility | POR (×10−2): 0.709 -> 0.687 | Helped |
Surprising findings
- The retrieval-based CAGE attains better object-level MMD and POR than the proposed generative model, which the authors attribute to superior reconstruction.
Limitations
- Limited dataset restricts application to few object types with few sub-parts; advantages for many parts (>10) are untested.
- Adding more object categories degrades SDF geometry reconstruction quality due to limited generalization.
- Fine-grained joint quantitative details (e.g., rotation angles) in text conditions are hard to capture.
Experimental setup
Datasets: PartNet PartNet-Mobility
Benchmarks: PartNet-Mobility (6 categories)
Metrics: MMD COV 1-NNA ID POR human study alignment (AL) human study diversity (DS)
Baselines: NAP-128 NAP-768 CAGE Ours-NAPSP Ours-PR
Builds on: Diffusion-SDF NAP CAGE DDPM Gumbel-Softmax
Closest papers by meaning
- SINGAPO: Single Image Controlled Generation of Articulated Parts in Objects (ICLR 2025 · similarity 0.90)
- CAGE: Controllable Articulation GEneration (CVPR 2024 · similarity 0.90)
- MeshArt: Generating Articulated Meshes with Structure-Guided Transformers (CVPR 2025 · similarity 0.90)
- DIPO: Dual-State Images Controlled Articulated Object Generation Powered by Diverse Data (NeurIPS 2025 · similarity 0.88)
- Real2Code: Reconstruct Articulated Objects via Code Generation (ICLR 2025 · similarity 0.87)
- Controllable Dynamic 3D Shape Generation via 3D Trajectories and Text (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.