Research map / Detection and segmentation / Open-set and few-shot detection
SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAM-6D, a two-stage zero-shot RGB-D framework that uses SAM proposals with a semantic/appearance/geometry matching score for instance segmentation, and a background-token point-matching pose estimator trained on synthetic data for 6D pose.
Problem
Zero-shot 6D object pose estimation: detecting all instances of novel, unseen objects in cluttered RGB-D scenes and estimating their 6D poses.
Approach
Use SAM to generate class-agnostic proposals, score them with a combined semantic, appearance and geometric matching score (ISM), then estimate poses via partial-to-partial point matching with learnable background tokens in a coarse-to-fine two-stage model with Sparse-to-Dense Point Transformers (PEM).
Key results
- Zero-shot pose estimation with SAM-6D (SAM) reaches mean AR 70.4 across the seven BOP core datasets, versus 62.2 for the best zero-shot ZeroPose† and 65.3 for GigaPose† style baselines on the BOP leaderboard comparison (Table 2).
- Instance segmentation mean mAP reaches 48.1 with SAM and all three matching scores, versus 40.4 for CNOS (SAM) (Table 1).
- On YCB-V, background tokens achieve AR 84.5 at 1.36 s per image versus 81.4 AR at 4.31 s for optimal transport (Table 3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add appearance and geometry matching scores to semantic matching (ablation) | SAM-6D instance segmentation · BOP seven core datasets (mean) | mAP: 44.0 -> 48.1 | Helped |
| Add appearance matching score to semantics and geometry matching (ablation) | SAM-6D instance segmentation · BOP seven core datasets (mean) | mAP: 46.7 -> 48.1 | Helped |
| Add geometry matching score to semantics and appearance matching (ablation) | SAM-6D instance segmentation · BOP seven core datasets (mean) | mAP: 45.0 -> 48.1 | Helped |
| Use SAM instead of FastSAM for proposal generation | SAM-6D instance segmentation · BOP seven core datasets (mean) | mAP: 44.9 -> 48.1 | Helped |
| Use background tokens instead of optimal transport (ablation) | PEM · YCB-V | AR: 81.4 -> 84.5 | Helped |
| Use background tokens instead of optimal transport (ablation) | PEM · YCB-V | inference time (s): 4.31 -> 1.36 | Helped |
| Add fine point matching after coarse point matching (ablation) | PEM · YCB-V | AR: 77.6 -> 84.5 | Helped |
| Add coarse point matching before fine point matching (ablation) | PEM · YCB-V | AR: 40.2 -> 84.5 | Helped |
| Use Sparse-to-Dense Point Transformers instead of Geometric Transformers (ablation) | PEM fine point matching · YCB-V | AR: 81.7 -> 84.5 | Helped |
| Use Sparse-to-Dense Point Transformers instead of Linear Transformers (ablation) | PEM fine point matching · YCB-V | AR: 78.4 -> 84.5 | Helped |
| Use SAM-6D instead of ZeroPose† for zero-shot detection and pose estimation | zero-shot pose estimation · BOP seven core datasets (mean) | AR: 51.2 -> 70.4 | Helped |
Surprising findings
- Removing the coarse point matching stage drops YCB-V AR from 84.5 to 40.2, showing the fine stage alone fails without an initial pose.
Limitations
- SAM-based variant is slower at pose estimation (4.37 s per image) due to more proposals.
- Performance on ITODD for instance segmentation and some datasets (e.g., T-LESS pose AR) trails certain baselines such as MegaPose† with RGB-D.
Experimental setup
Datasets: ShapeNet-Objects Google-Scanned-Objects BOP benchmark (LM-O, T-LESS, TUD-L, IC-BIN, ITODD, HB, YCB-V)
Benchmarks: BOP LM-O T-LESS TUD-L IC-BIN ITODD HB YCB-V
Metrics: mAP (IoU 0.50-0.95) mean Average Recall (VSD, MSSD, MSPD)
Baselines: ZeroPose CNOS MegaPose GigaPose OnePose++ Optimal transport
Builds on: Segment Anything Model CNOS Geometric Transformer DINOv2 MegaPose
Closest papers by meaning
- Any6D: Model-free 6D Pose Estimation of Novel Objects (CVPR 2025 · similarity 0.89)
- SingRef6D: Monocular Novel Object Pose Estimation with a Single RGB Reference (NeurIPS 2025 · similarity 0.88)
- Open-Vocabulary Object 6D Pose Estimation (CVPR 2024 · similarity 0.87)
- One2Any: One-Reference 6D Pose Estimation for Any Object (CVPR 2025 · similarity 0.87)
- Universal Features Guided Zero-Shot Category-Level Object Pose Estimation (AAAI 2025 · similarity 0.87)
- Pos3R: 6D Pose Estimation for Unseen Objects Made Easy (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.