Research map / Architectures and efficiency / Attention and transformers

LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision

NeurIPS 2025 · Architectures and efficiency › Attention and transformers · approach: Efficient and robust vision transformers · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LookWhere, a low-res selector plus high-res extractor pair of ViTs jointly pretrained by what-where distillation from DINOv2 so that high-resolution images are represented without full-resolution processing.

Problem

Vision transformers are expensive at high resolution because token count grows quadratically with image size; existing token reduction methods still process all tokens and token selection methods need complex per-task optimization.

Approach

A low-resolution ViT selector predicts a high-resolution patch importance map and a high-resolution ViT extractor processes only the top-k selected patches plus the selector's global tokens; both are pretrained jointly by distilling DINOv2's attention (where) and tokens (what), so the full high-res input is never processed.

Key results

• On ImageNet-1K, LookWhere ViT-B (k=128) reaches 83.0% top-1 at 14.8 GFLOPs versus DINOv2 ViT-B at 84.2% and 23.6 GFLOPs; ViT-S speed is 1.36x the second-fastest method (DTEM).
• On Swedish Traffic Signs (994px), LookWhere rivals IPS in accuracy (about 1.1% lower) while cutting inference time 6x and FLOPs 34x.
• On ADE20K, LookWhere (k=1024) reaches 44.6 mIoU at 32.8 GFLOPs versus DTEM's 44.3 mIoU at 25.8 GFLOPs at similar budget, and is more accurate than DTEM at 2x+ speed.

Measured effects reported in the paper

Surprising findings

• Random patch selection drops Traffic Signs accuracy by 25%, only marginally better than majority-class prediction, while the ImageNet drop is only 2.1%.
• LookWhere showed higher adversarial robust accuracy than its DINOv2 teacher under weak AutoAttack settings (0.75% vs 0.11% at eps 1/255), though this was not an adversarial training result.

Limitations

• The selector is not finetuned per task; task-specific selector finetuning could improve results.
• Design is spatial only; temporal or spectral extensions are untested.
• Accuracy is sometimes below the most accurate selection method (IPS is 1.1% higher on Traffic Signs with more computation).

Experimental setup

Datasets: ImageNet-1K ADE20K Swedish Traffic Signs CUB-200-2011 Billiard Balls ImageNet-HR AID NIH Chest X-ray

Benchmarks: ImageNet-1K ADE20K Swedish Traffic Signs CUB-200-2011 Billiard Balls ImageNet-HR AID NIH Chest X-ray

Metrics: top-1 accuracy mIoU FLOPs throughput (images/s) peak memory AUC-ROC robust accuracy under AutoAttack

Baselines: DINOv2 PiToMe ATC DTEM LTRP DPS IPS PatchDrop

Builds on: DINOv2 DPS IPS LTRP PatchDrop

Closest papers by meaning

• LookThere! Sparse Vision by Reinforced Selection (NeurIPS 2026 · similarity 0.92)
• LookWhen? Fast Video Recognition by Learning When, Where, and What to Compute (NeurIPS 2026 · similarity 0.90)
• SparseFormer: Sparse Visual Recognition via Limited Latent Tokens (ICLR 2024 · similarity 0.89)
• LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate (NeurIPS 2024 · similarity 0.87)
• Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials (NeurIPS 2025 · similarity 0.87)
• Register and [CLS] tokens induce a decoupling of local and global features in large ViTs (NeurIPS 2025 · similarity 0.86)