Research map / Detection and segmentation / Open-set and few-shot detection
Solving Instance Detection from an Open-World Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts pretrained foundation models (GroundingDINO/SAM proposals, DINOv2 features) for instance detection via metric learning with SAM-sampled distractors and NeRF novel-view augmentation, outperforming OTS-FM and VoxDet on HR-InsDet and RoboTools.
Problem
Instance detection requires localizing specific object instances in novel scenes from visual references; pretrained foundation models give high proposal recall but are not optimized for instance-level matching, and open-world distribution shifts and reference-proposal domain gaps persist.
Approach
Fine-tune a foundation feature model (DINOv2) with a metric learning triplet loss, using open-world data: SAM-generated distractor patches as negatives and per-instance NeRF (Zip-NeRF) novel-view synthesized references for both training and testing.
Key results
- IDOW with GroundingDINO reaches 57.01 AP on HR-InsDet CID, versus 51.68 AP for OTS-FM with GroundingDINO.
- IDOW with GroundingDINO reaches 59.4 AP on RoboTools NID, versus 56.7 AP for OTS-FM and 18.7 AP for VoxDet.
- Ablation: fine-tuning DINOv2 raises AP from 51.68 to 53.94, and NeRF-synthesized views used at test time give the largest gain (to 56.44 AP).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt the DINOv2 feature model through metric-learning fine-tuning (ablation) | OTS-FM GroundingDINO on HR-InsDet · HR-InsDet (CID) | AP: 51.68 -> 53.94 | Helped |
| Add NeRF-synthesized images during training (ablation) | Fine-tuned DINOv2 on HR-InsDet · HR-InsDet (CID) | AP: 53.94 -> 54.48 | Helped |
| Add distractor sampling (ablation) | Fine-tuned DINOv2 on HR-InsDet · HR-InsDet (CID) | AP: 53.94 -> 54.10 | Helped |
| Add NeRF-synthesized images during testing (ablation) | Fine-tuned DINOv2 on HR-InsDet · HR-InsDet (CID) | AP: 53.94 -> 55.24 | Helped |
| Use NeRF-synthesized images during testing rather than training (ablation) | Fine-tuned DINOv2 on HR-InsDet · HR-InsDet (CID) | AP: Train+DA@Train (54.48 AP) -> Train+DA@Test (55.24 AP) | Helped |
| Add NeRF-synthesized images during training and testing, plus distractor sampling (ablation) | Fine-tuned DINOv2 on HR-InsDet · HR-InsDet (CID) | AP: 51.68 -> 57.01 | Helped |
| Use more than 24 NeRF-generated images for training (ablation) | IDOW on HR-InsDet · HR-InsDet (CID) | AP | Hurt |
| Use more NeRF-generated images during testing (ablation) | IDOW on HR-InsDet · HR-InsDet (CID) | AP | Helped |
| Adapt the foundation model for instance-level matching | OTS-FM GroundingDINO on HR-InsDet · HR-InsDet (CID) | AP: 51.68 -> 57.01 | Helped |
| Adapt the foundation model for instance-level matching | OTS-FM GroundingDINO on RoboTools · RoboTools (NID) | AP: 56.7 -> 59.4 | Helped |
Surprising findings
- Using NeRF-synthesized novel-view images at test time brings more gain than using them in training, and adding more than 24 synthesized images per instance during training hurts performance.
Limitations
- Relies on GroundingDINO proposals, which are not yet real-time.
- Per-instance NeRF training is slow (about 1 hour per instance on a single 3090).
- Adding too many NeRF-synthesized training images causes overfitting to rendering artifacts.
Experimental setup
Datasets: HR-InsDet RoboTools OWID ShapeNet ABO
Benchmarks: HR-InsDet RoboTools
Metrics: AP AP50 AP75
Baselines: Cut-Paste-Learn OTS-FM VoxDet OS2D DTOID OLNCorr
Builds on: OTS-FM VoxDet Cut-Paste-Learn Zip-NeRF DINOv2
Closest papers by meaning
- Detecting Open World Objects via Partial Attribute Assignment (CVPR 2025 · similarity 0.86)
- v-CLR: View-Consistent Learning for Open-World Instance Segmentation (CVPR 2025 · similarity 0.86)
- Recognize Any Regions (NeurIPS 2024 · similarity 0.86)
- OW-OVD: Unified Open World and Open Vocabulary Object Detection (CVPR 2025 · similarity 0.86)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.86)
- OpenAD: Open-World Autonomous Driving Benchmark for 3D Object Detection (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.