Research map / 3D vision / 3D generation
Two by Two: Learning Multi-Task Pairwise Objects Assembly for Generalizable Robot Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces 2BY2, a dataset of 517 daily-object pairs across 18 pairwise assembly tasks with pose and symmetry annotations, and a two-step SE(3)-equivariant pose estimation network (two-scale VN-DGCNN) that outperforms shape-assembly baselines and transfers to a UR5 robot.
Problem
Existing assembly benchmarks focus on geometric fragments or factory parts, which do not capture everyday pairwise object assembly requiring semantic and spatial alignment; methods trained on them generalize poorly to such tasks.
Approach
Predict the pose of the base/receiving object (Object B) first, then predict the fitting object (Object A) conditioned on the transformed B, using a two-scale SE(3)-equivariant Vector Neuron DGCNN encoder with multiplicative cross-object feature fusion, trained stage-wise with ground-truth canonical B during training.
Key results
- Across 18 tasks on 2BY2, the method reduces translation RMSE and rotation RMSE on average by 0.046 and 8.97 versus the baselines; on ALL it reaches 0.110 translation and 41.44 rotation RMSE.
- Real-robot UR5 experiments on Cup, Flower, Bread, Plug achieve 77.5% overall success vs 22.5% for SE(3)-Assembly over 10 trials per task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the two-step pairwise network instead of SE(3)-Assembly | 2BY2 assembly pose estimation · ALL | RMSE(T): 0.233 -> 0.110 | Helped |
| use the two-step pairwise network instead of SE(3)-Assembly | 2BY2 assembly pose estimation · ALL | RMSE(R): 52.34 -> 41.44 | Helped |
| use the two-step pairwise network instead of SE(3)-Assembly | 2BY2 assembly pose estimation · Real-world Cup, Flower, Bread, Plug | success rate: 22.5% -> 77.5% | Helped |
| replace two-scale Vector Neuron DGCNN with Vector Neuron DGCNN (ablation) | two-step pairwise network · ALL | RMSE(T): 0.110 -> 0.123 | Hurt |
| replace two-scale Vector Neuron DGCNN with Vector Neuron DGCNN (ablation) | two-step pairwise network · ALL | RMSE(R): 41.44 -> 44.67 | Hurt |
| replace two-scale Vector Neuron DGCNN with DGCNN (ablation) | two-step pairwise network · ALL | RMSE(T): 0.110 -> 0.277 | Hurt |
| replace two-scale Vector Neuron DGCNN with DGCNN (ablation) | two-step pairwise network · ALL | RMSE(R): 41.44 -> 72.46 | Hurt |
| replace two-scale Vector Neuron DGCNN with PointNet (ablation) | two-step pairwise network · ALL | RMSE(T): 0.110 -> 0.264 | Hurt |
| replace two-scale Vector Neuron DGCNN with PointNet (ablation) | two-step pairwise network · ALL | RMSE(R): 41.44 -> 75.38 | Hurt |
| remove the two-step pose prediction and jointly predict both poses (ablation) | two-step pairwise network · ALL | RMSE(T): 0.110 -> 0.139 | Hurt |
| remove the two-step pose prediction and jointly predict both poses (ablation) | two-step pairwise network · ALL | RMSE(R): 41.44 -> 45.20 | Hurt |
Limitations
- Real-world evaluation limited to four tasks and 10 trials each
- Two-step training uses ground-truth canonical Object B during training, so errors in B's prediction are not seen during training
Experimental setup
Datasets: 2BY2
Benchmarks: 2BY2
Metrics: RMSE translation RMSE rotation (Euler angles, symmetry-aware) real-robot success rate
Baselines: SE(3)-Assembly Puzzlefusion++ Jigsaw Neural Shape Mating VN DGCNN DGCNN PointNet
Builds on: Vector Neuron networks DGCNN Neural Shape Mating
Closest papers by meaning
- Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Images (AAAI 2025 · similarity 0.88)
- BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly (ICML 2025 · similarity 0.87)
- IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos (NeurIPS 2024 · similarity 0.87)
- Category-Level Multi-Part Multi-Joint 3D Shape Assembly (CVPR 2024 · similarity 0.87)
- Imagine: Image-Guided 3D Part Assembly with Structure Knowledge Graph (AAAI 2025 · similarity 0.87)
- Generative 3D Part Assembly via Part-Whole-Hierarchy Message Passing (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.