Research map / 3D vision / 3D generation
Imagine: Image-Guided 3D Part Assembly with Structure Knowledge Graph
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Imagine, an image-guided 3D part assembly framework that builds a structure knowledge graph from a single image (via StructureNet-based reconstruction and GLIP visual features) and refines it jointly with coarse-to-fine part pose prediction.
Problem
3D part assembly (predicting 6-DoF poses of parts) struggles with complex assemblies and generalization to novel categories when methods memorize part poses; single-image structure cues are incomplete due to occlusion and view changes.
Approach
A reconstruction-based structure knowledge graph is extracted from the image (leaf parts with geometry, spatial center, semantic label) and used as an assembly prior; coarse relation modeling matches parts to graph nodes, fine relation modeling refines poses with geometric and image-semantic attention, and the graph is updated during assembly (knowledge-assembly co-evolution).
Key results
- On PartNet, Imagine reaches SCD 0.421 (chair), 0.252 (table), 0.235 (cabinet) vs. 0.534/0.343/0.368 for the best prior baseline 3DHPA.
- Chair-to-table generalization SCD 1.465 vs 1.839 (IET) and 2.103 (3DPA).
- Removing the visual feature f_vis lowers part accuracy from 65.88% to 62.05%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the structure knowledge graph (ablation) | Imagine · PartNet Chair | PA (%): 61.27 -> 65.88 | Helped |
| add the semantic-grounded visual feature fvis (ablation) | Imagine · PartNet Chair | PA (%): 62.05 -> 65.88 | Helped |
| add translation loss Lt (ablation) | Imagine · PartNet Chair | SCD (10−2): 0.655 -> 0.421 | Helped |
| add rotation loss Lr (ablation) | Imagine · PartNet Chair | PA (%): 63.09 -> 65.88 | Helped |
| add shape Chamfer distance loss Lscd (ablation) | Imagine · PartNet Chair | SCD (10−2): 0.481 -> 0.421 | Helped |
| add unmatched-part center supervision Lsp (ablation) | Imagine · PartNet Chair | PA (%): 64.25 -> 65.88 | Helped |
| replace GLIP features with pretrained tiny ViT features (ablation) | Imagine · PartNet Chair | SCD (10−2): 0.421 -> 0.436 | Hurt |
| replace GLIP features with 2D part segmentation features (ablation) | Imagine · PartNet Chair | SCD (10−2): 0.421 -> 0.449 | Hurt |
| use Imagine instead of 3DHPA | PartNet Chair · PartNet Chair | SCD (10−2): 0.534 -> 0.421 | Helped |
| use Imagine instead of 3DHPA | PartNet Table · PartNet Table | SCD (10−2): 0.343 -> 0.252 | Helped |
| use Imagine instead of 3DHPA | PartNet Cabinet · PartNet Cabinet | SCD (10−2): 0.368 -> 0.235 | Helped |
| use Imagine instead of IET | IKEA-Manual Chair · IKEA-Manual Chair | SCD (10−2): 1.097 -> 0.854 | Helped |
Surprising findings
- Generic pretrained ViT features nearly match GLIP visual features (PA 64.72% vs 65.88%), and 2D segmentation features give little improvement.
Limitations
- Fails on irregular parts' rotations and produces part collisions and disconnections.
- Single-view knowledge is insufficient for categories with large structural differences (cars, airplanes).
- Unseen-category generalization drops sharply (e.g., chair-to-cabinet).
Experimental setup
Datasets: PartNet IKEA-Manual Uy et al. 2021 rendered RGB images
Benchmarks: PartNet IKEA-Manual
Metrics: SCD PCD PA CA
Baselines: 3DPA IET 3DHPA DGL RGL
Builds on: StructureNet GLIP IET 3DPA PartSLIP
Closest papers by meaning
- Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Images (AAAI 2025 · similarity 0.91)
- Generative 3D Part Assembly via Part-Whole-Hierarchy Message Passing (CVPR 2024 · similarity 0.91)
- Category-Level Multi-Part Multi-Joint 3D Shape Assembly (CVPR 2024 · similarity 0.90)
- 3DCoMPaT200: Language Grounded Large-Scale 3D Vision Dataset for Compositional Recognition (NeurIPS 2024 · similarity 0.87)
- Two by Two: Learning Multi-Task Pairwise Objects Assembly for Generalizable Robot Manipulation (CVPR 2025 · similarity 0.87)
- Assembly Fuzzy Representation on Hypergraph for Open-Set 3D Object Retrieval (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.