Research map / 3D vision / 3D generation
Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces image-guided robotic assembly, where multi-view images of a brick model are translated into fine-grained assembly instructions (component types, poses, assembly order) by an end-to-end model, Neural Assembler, with an object graph.
Problem
Generating executable robotic assembly instructions (brick types, 3D poses, and physically feasible assembly order) from multi-view images of a structural 3D model, which requires 2D-3D correspondence under occlusion.
Approach
An end-to-end transformer-decoder model with CLIP multi-view fusion and library-based visual prompts (shape/texture templates) predicts brick masks, keypoints, rotations, and confidences per view, lifts keypoints to 3D via ray intersection, and predicts an assembly graph with a GCN whose edges are decoded into an order.
Key results
- On LEGO-Assembly, Neural Assembler reaches 22.9% per-scene accuracy and 43.9 complete rate vs. 2.5% and 25.8 for DETR3D.
- On CLEVR-Assembly, the full model achieves 22.5% per-scene accuracy and 41.5 complete rate, versus 8.0% for LSTM and 4.5% for DETR3D.
- On a real-world dataset after fine-tuning, Neural Assembler reaches 22.0% per-scene accuracy and 50.5 complete rate, above LSTM (7.3, 21.8) and DETR3D (2.4, 12.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Neural Assembler instead of DETR3D | LEGO-Assembly · LEGO-Assembly | Per-Step Acc: 41.7 -> 73.5 | Helped |
| use Neural Assembler instead of DETR3D | CLEVR-Assembly · CLEVR-Assembly | Per-Step Acc: 29.2 -> 69.2 | Helped |
| add scene consensus (ablation) | Neural Assembler · LEGO-Assembly | Per-scene Acc: 18.4 -> 22.9 | Helped |
| add scene consensus (ablation) | Neural Assembler · CLEVR-Assembly | Per-scene Acc: 6.6 -> 22.5 | Helped |
| add scene consensus (ablation) | Neural Assembler · LEGO-Assembly | F1: 0.829 -> 0.820 | Hurt |
| add scene consensus (ablation) | Neural Assembler · CLEVR-Assembly | F1: 0.781 -> 0.840 | Helped |
| use 3 views instead of 2 (ablation) | Neural Assembler · CLEVR-Assembly | Per-Step Acc: 56.1 -> 61.1 | Helped |
| use the full view setting instead of 3 views (ablation) | Neural Assembler · CLEVR-Assembly | Per-Step Acc: 61.1 -> 69.2 | Helped |
Surprising findings
- LSTM outperforms GCN-free baselines on some metrics but performs worse than GCN on order prediction, with the authors arguing time dependence is not crucial for assembly order.
Limitations
- Occlusion, especially of bricks under other bricks, still substantially degrades performance.
- Per-scene accuracy remains low (about 22-23%), so complete assembly is rarely achieved.
- Real-world evaluation required fine-tuning on synthetic data built from scanned real bricks.
Experimental setup
Datasets: CLEVR-Assembly LEGO-Assembly
Benchmarks: CLEVR-Assembly LEGO-Assembly Novel-Dataset Real-World Dataset
Metrics: Complete Rate Per-scene Acc Count Acc Order CR Per-step Acc Pos Acc Rot Acc Shape Acc Texture Acc mIoU
Baselines: LSTM DETR3D Ours (w/o consensus)
Builds on: CLIP FS-DETR CLIPSeg DETR3D PointNet
Closest papers by meaning
- Imagine: Image-Guided 3D Part Assembly with Structure Knowledge Graph (AAAI 2025 · similarity 0.91)
- Two by Two: Learning Multi-Task Pairwise Objects Assembly for Generalizable Robot Manipulation (CVPR 2025 · similarity 0.88)
- Generative 3D Part Assembly via Part-Whole-Hierarchy Message Passing (CVPR 2024 · similarity 0.87)
- Category-Level Multi-Part Multi-Joint 3D Shape Assembly (CVPR 2024 · similarity 0.87)
- IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos (NeurIPS 2024 · similarity 0.86)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.