Research map / Detection and segmentation / Open-set and few-shot detection
Interpreting Object-level Foundation Models via Visual Precision Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Visual Precision Search, a gradient-free, black-box submodular search over superpixel sub-regions that produces instance-level attribution maps for object-level foundation models (Grounding DINO, Florence-2).
Problem
Interpreting decisions of object-level foundation models (visual grounding and object detection) is hard: gradient-based attributions are corrupted by visual-text fusion and perturbation-based saliency maps are noisy.
Approach
Sparsify the image into SLIC superpixels and greedily rank them by a submodular objective combining a clue score (IoU-weighted confidence of the target box when only the sub-region is kept) and a collaboration score (drop in detection when the sub-region is removed), then derive attribution from marginal gains; no internal model parameters are used.
Key results
- On MS COCO, faithfulness Insertion for Grounding DINO rises from 0.4412 (D-RISE) to 0.5459 (ours), a 23.7% gain.
- For Florence-2 on RefCOCO, Insertion improves by 6.1% and Deletion by 66.9% over D-RISE, with Energy Point Game up 14.6%.
- On MS COCO misclassified samples with Grounding DINO, ESR reaches 69.73% vs 50.26% for D-RISE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Visual Precision Search instead of D-RISE | Grounding DINO · MS COCO | Insertion: 0.4412 -> 0.5459 | Helped |
| Use Visual Precision Search instead of D-RISE | Grounding DINO · RefCOCO | Insertion: 0.6178 -> 0.7419 | Helped |
| Use Visual Precision Search instead of D-RISE | Grounding DINO · LVIS V1 (rare) | Insertion: 0.2808 -> 0.3695 | Helped |
| Use Visual Precision Search instead of D-RISE | Florence-2 · MS COCO | Deletion: 0.0972 -> 0.0479 | Helped |
| Use Visual Precision Search instead of D-RISE | Florence-2 · RefCOCO | Deletion: 0.3505 -> 0.1159 | Helped |
| Use Visual Precision Search instead of D-RISE to explain grounding failures | Grounding DINO · RefCOCO (incorrect localization) | Insertion: 0.3486 -> 0.4981 | Helped |
| Use Visual Precision Search instead of D-RISE to explain misclassification | Grounding DINO · MS COCO (misclassified) | Insertion: 0.2170 -> 0.3357 | Helped |
| Use Visual Precision Search instead of D-RISE to explain misclassification | Grounding DINO · LVIS V1 (rare, misclassified) | Insertion: 0.1245 -> 0.1776 | Helped |
| Use Visual Precision Search instead of D-RISE to explain missed detections | Grounding DINO · MS COCO (undetected) | Insertion: 0.1538 -> 0.2102 | Helped |
| Use Visual Precision Search instead of D-RISE to explain missed detections | Grounding DINO · LVIS V1 (rare, undetected) | Insertion: 0.0703 -> 0.1155 | Helped |
| Add the Collaboration Score to the Clue Score (ablation) | Visual Precision Search · MS COCO | Insertion: 0.3632 -> 0.5459 | Helped |
| Add the Clue Score to the Collaboration Score (ablation) | Visual Precision Search · MS COCO | Insertion: 0.5370 -> 0.5459 | Helped |
Surprising findings
- Gradient-based methods such as ODAM perform poorly on Grounding DINO, attributed to text-visual fusion contaminating gradients.
- Increasing the number of sub-regions improves faithfulness but rapidly increases inference time.
Limitations
- Inference time grows rapidly with the number of sub-regions.
- MLLM-based models like Florence-2 lack confidence scores, so the search relies only on IoU.
- Theoretical guarantees hold only when the model outputs boxes and confidences without filtering low-confidence boxes.
Experimental setup
Datasets: MS COCO 2017 LVIS V1 RefCOCO
Benchmarks: MS COCO 2017 RefCOCO LVIS V1 (rare)
Metrics: Insertion AUC Deletion AUC Insertion (class) Deletion (class) Insertion (IoU) Deletion (IoU) average highest score Point Game Energy Point Game Explaining Successful Rate
Baselines: Grad-CAM SSGrad-CAM++ ODAM D-RISE D-HSIC
Builds on: D-RISE RISE Grad-CAM ODAM SLICO superpixels
Closest papers by meaning
- Beyond Accuracy: Ensuring Correct Predictions With Correct Rationales (NeurIPS 2024 · similarity 0.88)
- Probing the 3D Awareness of Visual Foundation Models (CVPR 2024 · similarity 0.88)
- General Object Foundation Model for Images and Videos at Scale (CVPR 2024 · similarity 0.88)
- VGent: Visual Grounding via Modular Design for Disentangling Reasoning and Prediction (CVPR 2026 · similarity 0.87)
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.87)
- GroundVLP: Harnessing Zero-Shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.