Research map / Vision-language and multimodal / Human and embodied modeling
Seeing More with Less: Human-like Representations in Vision Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies foveated (variable-resolution) sampling at fixed pixel budgets to pretrained LMMs and vision models, showing gains over uniform sampling on VQA and detection and emergent human-like attention and neuronal selectivity.
Problem
Large multimodal models process images at uniform resolution, wasting pixels on non-critical regions; it is unclear whether human-like foveated sampling improves accuracy and internal representations under tight pixel budgets.
Approach
Compares information-matched foveated versus uniform sampling (same number of pixels, interpolated back to original size) on existing pretrained LMMs without architectural changes, and analyzes the resulting attention distance and neuronal selectivity in a DETR/MDETR model trained with foveated inputs.
Key results
- Variable sampling at 3% pixel density improves BLIP-2 on SEED-Bench by 2.1% (48.6% vs 46.5% uniform), MDETR on GQA by 2.7% (46.8% vs 44.1%), and ViLT on VQAv2 reaches 64.9% vs 62.9% uniform, versus 81.1% full resolution.
- Models reach about 80% of full-resolution performance (e.g., ViLT on VQAv2) using only 3% of pixels.
- Foveated-trained MDETR shows larger attention distances in layers 2 and 3 and some ResNet101 filters are resolution-selective.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use foveated variable rather than uniform sampling at 3% density | ViLT · VQAv2 validation | accuracy: 62.9 -> 64.9 | Helped |
| Use foveated variable rather than uniform sampling at 3% density | MDETR · GQA | accuracy: 44.1 -> 46.8 | Helped |
| Use foveated variable rather than uniform sampling at 3% density | BLIP2 · GQA | accuracy: 40.7 -> 42.3 | Helped |
| Use foveated variable rather than uniform sampling at 3% density | BLIP2 · VQAv2 | accuracy: 56.2 -> 57.9 | Helped |
| Use foveated variable rather than uniform sampling at 3% density | InstructBLIP · VQAv2 | accuracy: 66.5 -> 66.4 | Hurt |
| Use foveated variable rather than uniform sampling at 3% density | LLaVA-v1.5 · VQAv2 | accuracy: 65.1 -> 65.9 | Helped |
| Use foveated variable rather than uniform sampling at 3% density | BLIP2 · SEED-Bench | total accuracy: 46.5% -> 48.6% | Helped |
| Shift variable-sampling fixation from center to bottom-left (ablation) | MDETR · GQA, 3% sample density | Total accuracy: 44.1 -> 45.2 | Helped |
| Shift variable-sampling fixation from center to bottom-right (ablation) | MDETR · GQA, 3% sample density | Total accuracy: 44.1 -> 45.4 | Helped |
| Shift variable-sampling fixation from center to top-left (ablation) | MDETR · GQA, 3% sample density | Total accuracy: 44.1 -> 45.0 | Helped |
| Shift variable-sampling fixation from center to top-right (ablation) | MDETR · GQA, 3% sample density | Total accuracy: 44.1 -> 45.5 | Helped |
| Use variable rather than uniform sampling for object detection | Vision-only models · COCO object detection | performance: up to 2.2% | Helped |
Surprising findings
- BLIP-2 with 3% variable sampling surpasses its full-resolution baseline on some SEED-Bench question types such as Instance Identity.
- Foveated sampling retains its advantage with corner fixations, indicating the gain is not solely due to center-biased photographer bias.
Limitations
- No architectural adaptation to foveated grids; only a single fixation point is used.
- Large LMMs were evaluated with pretrained checkpoints and not trained on foveated inputs; only DETR/MDETR was trained.
- Efficiency gains are not measured; the advantage is small (about 2%) and the reasons for some gains remain unexplained.
Experimental setup
Datasets: ImageNet COCO GQA VQAv2 SEED-Bench
Benchmarks: GQA SEED-Bench VQAv2 COCO
Metrics: accuracy average recall (AR) attention distance
Baselines: uniform sampling full-resolution baseline Mask R-CNN
Builds on: DETR MDETR BLIP-2 LLaVA ViLT
Closest papers by meaning
- LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Models (CVPR 2026 · similarity 0.88)
- CanViT: Toward Active-Vision Foundation Models (NeurIPS 2026 · similarity 0.86)
- CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception (ICML 2026 · similarity 0.86)
- Apollo: An Exploration of Video Understanding in Large Multimodal Models (CVPR 2025 · similarity 0.86)
- BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception (ICML 2026 · similarity 0.86)
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.