Research map / Vision-language and multimodal / Human and embodied modeling
Mind the GAP: Glimpse-based Active Perception improves generalization and sample efficiency of visual reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Glimpse-based Active Perception (GAP), which uses error-neuron saliency maps to sequentially glimpse salient image regions and feeds glimpse contents and locations to Transformer or Abstractor models for visual reasoning with better sample efficiency and OOD generalization.
Problem
Neural networks struggle to learn visual relations (same-different, abstract rules) from few samples and fail to generalize them to out-of-distribution object shapes, unlike humans who use active vision.
Approach
A saliency map from error neurons (local patch-difference) drives a winner-takes-all glimpse sequence with inhibition-of-return; glimpse contents ("what") and raw 2D glimpse locations ("where") are processed separately, with the Abstractor's relational cross-attention using only visual content to modulate attention over locations.
Key results
- GAP-Abstractor achieves state-of-the-art accuracy on SVRT with 500 and 1000 training samples, with particularly strong results on same-different tasks.
- On SVRT #1-OOD, GAP-Abstractor outperforms prior models across 13 out-of-distribution shape test sets.
- On the Super-CLEVR OOD test for CLEVR-ART ID task, Slot-Abstractor drops to 24.1% while GAP-Abstractor (IK) reaches 82.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GAP-Abstractor with error neurons on raw images instead of Slot-Abstractor | CLEVR-ART ID · Super-CLEVR test set | accuracy: 24.1 ± 0.8 -> 72.3 ± 2.3 | Helped |
| Apply error neurons to pre-trained ResNet-18 feature maps instead of raw images (ablation) | GAP-Abstractor · CLEVR-ART ID Super-CLEVR test set | accuracy: 72.3 ± 2.3 -> 80.2 ± 1.9 | Helped |
| Apply error neurons to pre-trained ResNet-18 feature maps instead of raw images (ablation) | GAP-Abstractor · CLEVR-ART ID original test set | accuracy: 93.4 ± 1.3 -> 92.7 ± 1.8 | No clear effect |
| Use the alternative IK saliency-map extractor instead of error neurons on raw images (ablation) | GAP-Abstractor · CLEVR-ART ID Super-CLEVR and original test sets | accuracy: 72.3 ± 2.3 -> 82.1 ± 2.2; 93.4 ± 1.3 -> 90.2 ± 2.1 | Mixed |
Surprising findings
- Slot-Abstractor fails almost completely (24.1%) on unseen Super-CLEVR objects despite high in-distribution accuracy.
- Transformer downstream performs worse than Abstractor, suggesting that implicitly mixing what and where content weakens generalization.
Limitations
- Evaluated only on simple synthetic binary visual reasoning datasets with 2D shapes and limited realistic images.
- Saliency map from error neurons may be distracted by spurious salient locations such as shadows or reflections in realistic scenes.
- No consistent quantitative advantage between multi-scale and log-polar glimpse sensors.
Experimental setup
Datasets: SVRT SVRT #1-OOD ART CLEVR-ART Super-CLEVR
Benchmarks: SVRT SVRT #1-OOD ART CLEVR-ART
Metrics: accuracy
Baselines: Transformer Abstractor GAMR ResNet Attn-ResNet OCRA Slot-Abstractor
Builds on: Abstractor Transformer Slot-Abstractor GAMR
Closest papers by meaning
- DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning (ICLR 2026 · similarity 0.90)
- I-Perceive: A Foundation Model for Vision-Language Active Perception (NeurIPS 2026 · similarity 0.89)
- BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perception (NeurIPS 2025 · similarity 0.89)
- The Perceptual Bandwidth Bottleneck in Vision-Language Models: Active Visual Reasoning via Sequential Experimental Design (ICML 2026 · similarity 0.88)
- Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects (NeurIPS 2024 · similarity 0.88)
- From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.