Research map / Architectures and efficiency / Attention and transformers
Frequency-Aware Token Reduction for Efficient Vision Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a frequency-aware token reduction for ViTs that keeps high-frequency tokens (selected via attention high-pass component) and aggregates low-frequency tokens into local DC tokens, mitigating rank collapse while cutting compute.
Problem
Token reduction in Vision Transformers lowers compute but ignores the frequency characteristics of self-attention, where rank collapse and over-smoothing discard high-frequency token information and accelerate degradation.
Approach
Split tokens into high-frequency (HF) and low-frequency (LF) sets using a high-pass component of the attention map (cheap column-wise averaging), keep HF tokens, merge LF tokens into local DC tokens, and add learnable attention reweighting (omega terms) to counter collapse in pretrained models.
Key results
- On DeiT-S, the method reaches 79.8% top-1 at 3.0 GMACs, matching the 79.8% baseline at 4.6 GMACs (Table 1).
- On DeiT-T, the method reaches 72.3% at 0.8 GMACs versus 72.2% baseline at 1.3 GMACs.
- On ADE20K with Segmenter, 30% pruning on layers 4,7,10 gives 50.8 mIoU vs 51.3 baseline with throughput rising from 21.7 to 92.1 img/s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use frequency-aware token reduction instead of the unpruned model | ViT-S · ImageNet-1K | Top-1 accuracy (%): 78.8 -> 79.0 | Helped |
| Use frequency-aware token reduction instead of the unpruned model | ViT-S-21K · ImageNet-1K | Top-1 accuracy (%): 81.1 -> 81.2 | Helped |
| Use frequency-aware token reduction instead of the unpruned model | ViT-B-MAE · ImageNet-1K | Top-1 accuracy (%): 83.7 -> 83.2 | Hurt |
| Use frequency-aware token reduction instead of the unpruned model | ViT-B-DINO · ImageNet-1K | Top-1 accuracy (%): 81.9 -> 81.4 | Hurt |
| Retain HF tokens rather than pruning them (ablation) | Frequency-aware token reduction · ImageNet-1K | accuracy | Helped |
| Add a DC token to preserve low-frequency information (ablation) | Frequency-aware token reduction · ImageNet-1K | accuracy | Helped |
| Use local DC tokens instead of a global DC token (ablation) | Frequency-aware token reduction · ImageNet-1K, single-layer reduction | accuracy | Mixed |
| Enable ω1 attention reweighting (ablation) | DeiT-S frequency-aware token reduction · ImageNet-1K | Top-1 accuracy (%): 79.8% -> 79.9% | Helped |
| Add ω2 attention reweighting (ablation) | DeiT-S frequency-aware token reduction · ImageNet-1K fine-tuning | epochs required | Helped |
| Apply frequency-aware token reduction with 30% pruning on layers 4, 7, and 10 | Segmenter with ViT-B (EVA) · ADE20K | mIoU (%): 51.3 -> 50.8 | Hurt |
| Use frequency-aware token reduction | ViT-B · ImageNet-1K, 16/384 patch/image size | Top-1 accuracy (%): 86.6% to 87.0% | Helped |
| Vary the local window size for DC aggregation (ablation) | Frequency-aware token reduction · ImageNet-1K, single-layer reduction | accuracy | Mixed |
Surprising findings
- The pruned model sometimes exceeds the unpruned baseline in accuracy (e.g., DeiT-T 72.3 vs 72.2, ViT-B-21K-style gains), which the authors attribute to mitigating rank collapse.
- Removing HF tokens hurts accuracy more than removing LF tokens, and an LF-only retention setting drops to nearly 20% accuracy during fine-tuning.
Limitations
- Experiments limited to vision models; applicability to multimodal or language Transformers is untested.
- Requires fine-tuning (30 epochs) of pretrained models rather than being training-free.
- Some reported configurations trade accuracy for compute (e.g., Ours-o2 on DeiT-S drops to 73.5%).
Experimental setup
Datasets: ImageNet-1K ADE20K
Benchmarks: ImageNet-1K ADE20K
Metrics: top-1 accuracy MACs parameters throughput mIoU
Baselines: ToMe EViT DynamicViT Evo-ViT ATS METR DTEM Zero-TP DiffRate
Builds on: ToMe EViT DiffRate DeiT
Closest papers by meaning
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.88)
- Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers (CVPR 2024 · similarity 0.87)
- An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.87)
- ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition (AAAI 2025 · similarity 0.87)
- MLP Can Be A Good Transformer Learner (CVPR 2024 · similarity 0.86)
- Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.