Research map / Architectures and efficiency / Attention and transformers
LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
LF-ViT runs a ViT on a down-sampled image and, when confidence is low, uses a Neighborhood Global Class Attention mechanism to select class-discriminative regions from the full-resolution image for a second focus stage, sharing parameters across both stages.
Problem
Vision Transformers incur high computation and memory costs on high-resolution images due to spatial redundancy, where most image regions are not task-relevant for recognition.
Approach
A two-stage localization-and-focus inference with shared parameters: early exit on a half-resolution image, and otherwise NGCA-based selection of the most class-discriminative region in the original image, plus reuse and fusion of token features across stages, trained end-to-end with CE plus KL loss.
Key results
- LF-ViT matches DeiT-S accuracy (79.8%) at 1.7G FLOPs (63% reduction) with 2.03x throughput on A100.
- LF-ViT(m=6, eta=0.85) reaches 81.0% top-1 at 3.0G FLOPs, above DeiT-S's 79.8%.
- LF-ViT* (288 input) reaches 82.2% at 3.7G FLOPs, outperforming CF-ViT* (81.9%, 4.8G).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LF-ViT localization and focus inference instead of full-resolution DeiT-S inference | DeiT-S · ImageNet val | FLOPs (G): 4.6 -> 1.7 | Helped |
| Use LF-ViT localization and focus inference instead of full-resolution DeiT-S inference | DeiT-S · ImageNet val | top-1 accuracy: 79.8 -> 79.8 | No clear effect |
| Increase η from 0.46 to 0.85 with m=6 | LF-ViT · ImageNet val | top-1 accuracy: 79.8(+0.0) -> 81.0(+1.2) | Helped |
| Replace NGCA with negative NGCA region selection (ablation) | LF-ViT region identification · ImageNet val, focus stage | top-1 accuracy: 80.8 -> 79.7 | Hurt |
| Replace NGCA with maximum GCA region selection (ablation) | LF-ViT region identification · ImageNet val, focus stage | top-1 accuracy: 80.8 -> 80.2 | Hurt |
| Replace NGCA with minimum GCA region selection (ablation) | LF-ViT region identification · ImageNet val, focus stage | top-1 accuracy: 80.8 -> 79.8 | Hurt |
| Replace NGCA with random region selection (ablation) | LF-ViT region identification · ImageNet val, focus stage | top-1 accuracy: 80.8 -> 80.0 | Hurt |
| Replace localization-stage KL supervision with CE supervision (ablation) | LF-ViT training · ImageNet val, focus stage | top-1 accuracy: 80.8 -> 80.4 | Hurt |
| Use β=0.0 instead of the default β=0.99 (ablation) | LF-ViT GCA calculation · ImageNet val, focus stage | top-1 accuracy: 80.8 -> 80.5 | Hurt |
| Use β=0.5 instead of the default β=0.99 (ablation) | LF-ViT GCA calculation · ImageNet val, focus stage | top-1 accuracy: 80.8 -> 80.6 | Hurt |
| Use β=0.9 instead of the default β=0.99 (ablation) | LF-ViT GCA calculation · ImageNet val, focus stage | top-1 accuracy: 80.8 -> 80.7 | Hurt |
| Use β=0.999 instead of β=0.99 (ablation) | LF-ViT GCA calculation · ImageNet val, focus stage | top-1 accuracy: 80.8 -> 80.8 | No clear effect |
Surprising findings
- Higher confidence threshold eta improves accuracy above the DeiT-S baseline (up to +1.2%) at the cost of more FLOPs.
Limitations
- Relies on a hand-tuned confidence threshold eta and region size m, trading accuracy against computation.
- Evaluated only on ImageNet classification with DeiT-S backbone.
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet
Metrics: top-1 accuracy FLOPs throughput (img/s)
Baselines: DeiT-S DynamicViT IA-RED$^2$ PS-ViT EViT Evo-ViT A-ViT-S PVT-S SaiT-S CF-ViT DVT T2T-ViT
Builds on: DeiT ViT CF-ViT DVT GFNet
Closest papers by meaning
- RegionViT: Regional-to-Local Attention for Vision Transformers (ICLR 2022 · similarity 0.89)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.89)
- Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice (ICLR 2022 · similarity 0.88)
- QT-ViT: Improving Linear Attention in ViT with Quadratic Taylor Expansion (NeurIPS 2024 · similarity 0.87)
- RPFQ-ViT: Rotated Phase-Frame Quantization for Extremely Low-Bit Weights in Vision Transformers (NeurIPS 2026 · similarity 0.87)
- Attention Guided CAM: Visual Explanations of Vision Transformer Guided by Self-Attention (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.