Research map / Architectures and efficiency / Attention and transformers

LookThere! Sparse Vision by Reinforced Selection

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Efficient and robust vision transformers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LookThere, an end-to-end RL framework where a shallow low-resolution selector learns which high-resolution ViT patches to process jointly with a deep extractor, achieving better accuracy-compute trade-offs at extreme sparsity than LookWhere and other token selection/reduction methods.

Problem

Adaptive computation in vision transformers processes many uninformative tokens; existing token selection methods rely on attention/diversity heuristics or distillation and struggle at extreme sparsity on high-resolution inputs.

Approach

Jointly train a shallow selector (policy over hierarchical Bernoulli patch masks, computed on a low-res view) and a deep ViT extractor, optimizing the selector with GRPO against a reward combining task performance and a gated sparsity term, instead of distilling attention or using proxy objectives.

Key results

• On Billiards at 99% sparsity, LookThere reaches 94.3% accuracy vs 20.8% for IPS and 60.1% for DPS (Table in appendix; LookWhere degrades to ~30% at 99.7%).
• On Birdsnap with DINOv3, LookThere reaches 96.4/96.7/86.8% at 90/95/99% sparsity (approximate; as stated in text).
• Achieves up to >10x throughput and >5x fewer FLOPs than merging/pruning at high resolution, with Birdsnap at 90% sparsity 87.1% accuracy at 15 GFLOPs vs DINOv2 dense 90.0% at 120 GFLOPs.

Surprising findings

• DINOv2 attention-based selection scatters attention to visually salient but task-irrelevant regions, so attention-distilled selection can fail on the downstream task.
• Gradually annealing sparsity targets gave slightly lower sparsity than fixed targets, and hierarchical vs flat policies reach comparable accuracy.

Limitations

• RL inherits sample inefficiency and may overfit in low-data regimes where the extractor compensates for poor selection.
• Reward assigned to full mask makes patch-level credit assignment difficult.
• Selection only studied across space, not time or modality.

Experimental setup

Datasets: ImageNet-1K ADE20K FSC-147 Traffic Signs (TT100K) Billiard Balls Birdsnap Places365 MS-COCO

Benchmarks: ImageNet-1K ADE20K Traffic Signs (TT100K) Billiard Balls Birdsnap Places365 MS-COCO FSC-147

Metrics: top-1 accuracy mIoU mask AP MAE FLOPs throughput sparsity

Baselines: LookWhere DPS IPS DTEM Token Cropr PiToMe ATC REINFORCE MaxRL

Builds on: LookWhere GRPO PatchDrop DPS DINOv2

Closest papers by meaning

• LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision (NeurIPS 2025 · similarity 0.92)
• SparseFormer: Sparse Visual Recognition via Limited Latent Tokens (ICLR 2024 · similarity 0.88)
• LookWhen? Fast Video Recognition by Learning When, Where, and What to Compute (NeurIPS 2026 · similarity 0.86)
• Beyond Interpretability: When, Why, and How Sparse Autoencoders Enable Label-Free Visual Steering (NeurIPS 2026 · similarity 0.86)
• CanViT: Toward Active-Vision Foundation Models (NeurIPS 2026 · similarity 0.86)
• CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models (ICML 2025 · similarity 0.86)