Research map / Video / Video segmentation and tracking
IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces IKEA Video Manuals, a dataset aligning 3D furniture parts, IKEA manuals, and Internet assembly videos with dense 2D masks and 6-DoF poses, and benchmarks five grounding tasks with off-the-shelf models.
Problem
Existing shape-assembly datasets do not provide 4D (3D over time) grounding of assembly instructions in Internet videos, linking manuals, 3D part models, and video frames.
Approach
A multimodal dataset built on Internet IKEA assembly videos with temporal step/substep alignment to manuals, per-frame part segmentation masks, and 6-DoF part poses from 2D-3D correspondences with PnP/RANSAC and manual refinement, plus baseline evaluations on five tasks.
Key results
- Part-conditioned segmentation: SAM-6D reaches 0.16 IoU and 0.40 Top-5 IoU versus 0.09 and 0.21 for CNOS, both low on the dataset.
- Part-conditioned pose estimation: MegaPose achieves the best ADD of 1.36 and ADD-S of 0.89, and all methods show high errors, indicating difficulty.
- Video object segmentation: SAM2 Hiera-L scores 73.6 J&F on the dataset versus 91.6 on DAVIS 2017, and Cutie-base drops to 54.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use geometric features to cluster parts with GeoCluster instead of the SingleStep heuristic | Assembly plan generation · IKEA Video Manuals | Simple Matching F1: 26.88 -> 29.74 | Helped |
| Generate plans from video-derived assembly sequences rather than IKEA-Manual plans | SingleStep and GeoCluster · IKEA Video Manuals vs IKEA-Manual | Simple Matching F1: 48.64 -> 26.88; 43.53 -> 29.74 | Hurt |
| Use SAM-6D instead of CNOS for part-conditioned segmentation | Part-conditioned segmentation · IKEA Video Manuals | IoU: 0.09 -> 0.16 | Helped |
| Use MegaPose instead of SAM-6D for part-conditioned pose estimation | Part-conditioned pose estimation · IKEA Video Manuals | ADD: 2.34 -> 1.36 | Helped |
| Use the occlusion-aware silhouette re-projection loss instead of MSE loss | Differentiable rendering pose estimation · IKEA Video Manuals | ADD: 3.33 -> 3.29 | Helped |
| Evaluate video object segmentation on IKEA Video Manuals rather than existing benchmarks | SAM2 Hiera-L · IKEA Video Manuals vs SA-V | J&F: 75.6 -> 73.6 | Hurt |
| Evaluate video object segmentation on IKEA Video Manuals rather than existing benchmarks | Cutie-base · IKEA Video Manuals vs SA-V | J&F: 60.7 -> 54.7 | Hurt |
| Use GPT-4o-detected keyframes instead of annotated keyframes | Video-based shape assembly pipeline · IKEA Video Manuals | Chamfer Distance: 0.33 -> 0.55 | Hurt |
| Use GPT-4o to identify keyframes | Video-based shape assembly pipeline · 15 IKEA Video Manuals videos | Final assembly completion: 5 of the 15 videos | Hurt |
Surprising findings
- GPT-4o fails to identify the final assembly step in 5 of 15 videos in the instruction-video shape assembly setting.
Limitations
- Dataset size is limited (98 videos, 36 furniture models), preventing large-scale training.
- Annotation requires manual effort, limiting scale-up.
- Only visual and 3D modalities; no audio or text.
Experimental setup
Datasets: IKEA-Manual IAW SA-V MOSE DAVIS 2017 LVOS YouTube-VOS 2019
Benchmarks: IKEA Video Manuals IKEA-Manual SA-V MOSE DAVIS 2017 LVOS YouTube-VOS 2019
Metrics: Precision Recall F1 IoU Top-5 IoU ADD ADD-S J&F Chamfer Distance
Baselines: SingleStep GeoCluster CNOS MegaPose Cutie Differentiable rendering
Builds on: Segment Anything Model IKEA-Manual IAW
Closest papers by meaning
- Two by Two: Learning Multi-Task Pairwise Objects Assembly for Generalizable Robot Manipulation (CVPR 2025 · similarity 0.87)
- Imagine: Image-Guided 3D Part Assembly with Structure Knowledge Graph (AAAI 2025 · similarity 0.86)
- Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Images (AAAI 2025 · similarity 0.86)
- VideoCAD: A Dataset and Model for Learning Long‑Horizon 3D CAD UI Interactions from Video (NeurIPS 2025 · similarity 0.84)
- Category-Level Multi-Part Multi-Joint 3D Shape Assembly (CVPR 2024 · similarity 0.84)
- Generative 3D Part Assembly via Part-Whole-Hierarchy Message Passing (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.