Research map / Detection and segmentation / Open-set and few-shot detection
Open-Vocabulary Object 6D Pose Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces open-vocabulary object 6D pose estimation where a text prompt specifies the object, and proposes Oryon, a CLIP-based fusion model that segments the object and matches features across two RGBD scenes, followed by PointDSC registration.
Problem
Estimating the relative 6D pose of an object seen in two different RGBD scenes without an object model or video at inference, with the object specified only by a textual prompt.
Approach
Fuses CLIP text embeddings with CLIP visual feature maps via a cost-volume and cost-aggregation transformer (guided by a Swin backbone), jointly trained for segmentation (Dice loss) and dense feature matching (contrastive loss), then registers back-projected matches with PointDSC.
Key results
- On REAL275, Oryon reaches 32.2 AR with its own masks versus 24.4 for SIFT and 22.4 for ObjectMatch (with its masks).
- On Toyota-Light, Oryon reaches 30.3 AR with its own masks, versus 27.2 for SIFT and 8.3 for ObjectMatch.
- Replacing the textual object name with 'object' drops REAL275 AR to 3.0 with predicted masks, showing reliance on the prompt.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Oryon instead of SIFT + PointDSC | Relative pose estimation with predicted masks · REAL275 | AR: 24.4 -> 32.2 | Helped |
| use Oryon instead of ObjectMatch | Relative pose estimation with predicted masks · REAL275 | AR: 22.4 -> 32.2 | Helped |
| use Oryon instead of SIFT + PointDSC | Relative pose estimation with predicted masks · Toyota-Light | AR: 27.2 -> 30.3 | Helped |
| use Oryon instead of ObjectMatch | Relative pose estimation with predicted masks · Toyota-Light | AR: 8.3 -> 30.3 | Helped |
| add guidance features to the fusion module (ablation) | Oryon · REAL275 | AR: 24.8 -> 32.2 | Helped |
| add guidance features to the decoder (ablation) | Oryon · REAL275 | AR: 21.2 -> 32.2 | Helped |
| add the extra upsampling layer (ablation) | Oryon · REAL275 | AR: 25.1 -> 32.2 | Helped |
| use PointDSC instead of RANSAC for registration (ablation) | Oryon · REAL275 | AR: 15.0 -> 32.2 | Helped |
| jointly train segmentation and feature matching instead of training segmentation only (ablation) | Oryon · REAL275 | AR: 24.8 -> 32.2 | Helped |
| jointly train segmentation and feature matching instead of training feature matching only (ablation) | Oryon · REAL275 | AR: 40.3 -> 46.5 | Helped |
| replace the object name with “object” in the prompt (ablation) | Oryon · REAL275 | AR: 32.2 -> 3.0 | Hurt |
| use a misleading brief description in the prompt (ablation) | Oryon · REAL275 | AR: 32.2 -> 25.4 | Hurt |
Surprising findings
- With uninformative prompts ('object'), segmentation completely fails (mIoU 3.0), while removing only the description keeps AR within about 2 points of the baseline.
- ObjectMatch performs much worse than hand-crafted SIFT on Toyota-Light due to lighting domain shift.
Limitations
- Requires depth maps for registration, limiting use to RGBD settings.
- Describing some objects (e.g., mechanical components) via text prompt may be hard.
- Large gap between predicted-mask and oracle-mask performance on REAL275 (about 14 AR).
Experimental setup
Datasets: ShapeNet6D REAL275 Toyota-Light
Benchmarks: REAL275 Toyota-Light
Metrics: AR VSD MSSD MSPD ADD(S)-0.1d mIoU
Baselines: SIFT ObjectMatch OVSeg + SIFT OVSeg + ObjectMatch
Builds on: CLIP PointDSC ObjectMatch SIFT OVSeg
Closest papers by meaning
- Any6D: Model-free 6D Pose Estimation of Novel Objects (CVPR 2025 · similarity 0.89)
- SingRef6D: Monocular Novel Object Pose Estimation with a Single RGB Reference (NeurIPS 2025 · similarity 0.88)
- One2Any: One-Reference 6D Pose Estimation for Any Object (CVPR 2025 · similarity 0.88)
- FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects (CVPR 2024 · similarity 0.88)
- SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation (CVPR 2024 · similarity 0.87)
- CapeX: Category-Agnostic Pose Estimation from Textual Point Explanation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.