Research map / Detection and segmentation / Open-set and few-shot detection
Find your Needle: Small Object Image Retrieval via Multi-Object Attention Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaO, a multi-object image retrieval framework for small object retrieval that fine-tunes a visual backbone on averaged object-crop embeddings and then optimizes a single unified descriptor to align explainability maps with object masks, plus three new SoIR benchmarks.
Problem
Retrieving images containing a specific small object in cluttered scenes with many distractor objects, using a single global descriptor that must represent all objects in the image.
Approach
Decompose images into OVD-detected object crops, encode each with a ViT backbone, and in a post-training stage optimize a single token so its LeGrad explainability maps across all crops align with SAM object masks (extending MaskInversion to multiple objects).
Key results
- On VoxDet zero-shot, MaO-DINOv2 reaches 70.20 mAP vs. 51.23 for DINOv2 and 52.01 for GSS.
- After fine-tuning, MaO-DINOv2 reaches 83.70 mAP on VoxDet and MaO-CLIP reaches 91.29 on INSTRE-XS vs. 72.90 for fine-tuned CLIP.
- On ILIAS (no distractors), zero-shot MaO-SigLIP-B reaches 62.22 mAP vs. 20.20 for SigLIP-B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply MaO attention optimization to CLIP | CLIP · VoxDet | mAP: 44.52 -> 65.22 | Helped |
| apply MaO attention optimization to DINOv2 | DINOv2 · VoxDet | mAP: 51.23 -> 70.20 | Helped |
| apply MaO to DINOv2 | DINOv2 · PerMiR_W | mAP: 40.57 -> 43.46 | Helped |
| fine-tune MaO-DINOv2 on VoxDet | MaO-DINOv2 · PerMiR_W | mAP: 43.46 -> 40.44 | Hurt |
| fine-tune MaO-CLIP on VoxDet | MaO-CLIP · VoxDet | mAP: 65.22 -> 79.86 | Helped |
| add MaO multi-object optimization to SIGLIP-B | SIGLIP-B · ILIAS | mAP: 20.20 -> 62.22 | Helped |
| add MaO to CLIP | CLIP fine-tuned · PODS | mAP: 0.341 -> 0.368 | Helped |
| add MaO to DINOv2 | DINOv2 fine-tuned · PODS | mAP: 0.332 -> 0.393 | Helped |
| replace Qwen2.5-VL pair matching with MaO single-vector retrieval | VoxDet subset · VoxDet subset | accuracy: 19% -> 85% | Helped |
| use Stage A multi-object fine-tuning without Stage B optimization (ablation) | MaO-DINOv2-B · ILIAS | mAP: 45.29 -> 48.85 | Hurt |
| replace multi-object crop optimization with full-image optimization (ablation) | fine-tuned DINOv2 · VoxDet_W | mAP | Hurt |
| increase scene clutter by adding objects to gallery representations (ablation) | MaO · VoxDet | mAP: 0.96 -> approximately 0.82 | Hurt |
Surprising findings
- Naively optimizing a single attention map over the full image lowers performance on VoxDet-W, showing that per-object crop optimization is needed.
- Qwen2.5-VL pairwise matching reaches only 19% accuracy on a VoxDet subset, versus 85% for MaO-DINOv2 single-vector retrieval.
Limitations
- Performance depends on open-vocabulary detector recall and localization; partial detections cause retrieval failures.
- Under heavy clutter (more than about 25 detected objects) the target can be under-represented and missed.
- Higher offline gallery encoding cost due to detection and per-object optimization; weaker on faces, individuals and landmarks.
Experimental setup
Datasets: VoxDet INSTRE PerMiR ILIAS PODS
Benchmarks: VoxDet-SoIR PerMiR INSTRE-XS INSTRE-XXS ILIAS PODS
Metrics: mAP AP recall runtime (ms)
Baselines: GSS CVNet SuperGlobal AMES GeM CLIP DINOv2 alpha-CLIP
Builds on: MaskInversion LeGrad OWLv2 SAM InfoNCE
Closest papers by meaning
- ILIAS: Instance-Level Image retrieval At Scale (CVPR 2025 · similarity 0.87)
- ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity (ICLR 2022 · similarity 0.87)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.86)
- Efficient Object Instance Search Using Fuzzy Objects Matching (AAAI 2017 · similarity 0.86)
- MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions (ICML 2024 · similarity 0.84)
- Dual Pose-invariant Embeddings: Learning Category and Object-specific Discriminative Representations for Recognition and Retrieval (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.