Research map / Vision-language and multimodal / Human and embodied modeling
ScanFormer: Referring Expression Comprehension by Iteratively Scanning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ScanFormer, a coarse-to-fine iterative perception unified Transformer for referring expression comprehension that discards linguistically irrelevant image patches across an image scale pyramid using learned selection and constant-token merging.
Problem
Dense perception of images in referring expression comprehension (REC) processes many visual regions unrelated to the language query, adding computational overhead, and existing efficient ViT methods focus on vision tasks rather than vision-language grounding.
Approach
Iteratively scan an image scale pyramid from coarse to fine, learn per-patch selection factors that are replaced by a learnable constant token, and merge the constant tokens in attention (log(N_c) key trick) so discarded patches cost little compute; cache keys/values across scales and regress a box via a [REG] token at each scale.
Key results
- Achieves 83.40 / 85.86 / 78.81 on RefCOCO val/testA/testB and 74.10 on RefCOCOg val, comparable to QRNet (Swin-S) while running over twice as fast.
- Real-time inference at 34.9 FPS on a 1080 Ti.
- On average 270 of 400 tokens are used per RefCOCOg sample; roughly 40 and 220 tokens per sample are merged in scales 1 and 2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add large-scale pre-training to ScanFormer | ScanFormer · RefCOCO testA | Acc@0.5: 85.86 -> 89.99 | Helped |
| Add large-scale pre-training to ScanFormer | ScanFormer · RefCOCO testB | Acc@0.5: 78.81 -> 82.89 | Helped |
| Add large-scale pre-training to ScanFormer | ScanFormer · RefCOCO+ testA | Acc@0.5: 77.57 -> 84.04 | Helped |
| Add large-scale pre-training to ScanFormer | ScanFormer · RefCOCO+ testB | Acc@0.5: 62.50 -> 70.63 | Helped |
| Add large-scale pre-training to ScanFormer | ScanFormer · RefCOCOg test | Acc@0.5: 74.14 -> 82.75 | Helped |
| Use the finer image-pyramid scales after the coarsest scale (ablation) | ScanFormer · RefCOCOg val | IoU | Helped |
Surprising findings
- Predicted early-exit metrics (IoU, GIoU, box variance) correlated poorly with localization accuracy, so early exit was left unresolved.
Limitations
- Localizes through all scales with no flexible early exit mechanism.
- Predicts only one target object per query, so it does not extend to phrase grounding.
- Early-exit metrics tried (IoU, GIoU, variance) did not correlate well with accuracy.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg ReferItGame Visual Genome regions Flickr entities
Benchmarks: RefCOCO RefCOCO+ RefCOCOg ReferItGame
Metrics: Acc@0.5 FPS
Baselines: MAttNet RefTR QRNet SeqTR LADS Word2Pix PLV-FPN MDETR OFA
Builds on: ViLT TransVG DVT CF-ViT improved semantic hashing
Closest papers by meaning
- RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.91)
- Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks (ICLR 2026 · similarity 0.91)
- Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point (AAAI 2024 · similarity 0.89)
- Referencing Where to Focus: Improving Visual Grounding with Referential Query (NeurIPS 2024 · similarity 0.89)
- DiffusionREC: Diffusion Model with Adaptive Condition for Referring Expression Comprehension (AAAI 2025 · similarity 0.89)
- A Large-Scale Human-Centric Benchmark for Referring Expression Comprehension in the LMM Era (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.