Research map / Architectures and efficiency / Attention and transformers
Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Tri-Level E-ViT at keep ratio 0.9 | DeiT-T · ImageNet-1K val | Top-1 accuracy: 72.2 -> 72.6 (+0.4) | Helped |
| Apply Tri-Level E-ViT at keep ratio 0.9 | DeiT-S · ImageNet-1K val | Top-1 accuracy: 79.8 -> 79.9 (+0.1) | Helped |
| Apply Tri-Level E-ViT at keep ratio 0.9 | Swin-T · ImageNet-1K val | training time reduced: 13.1% | Helped |
| Apply Tri-Level E-ViT | PiT-S · ImageNet-1K | training time reduced: 19.3% | Helped |
| Replace informative-example selection with random subset updates (ablation) | DeiT-S example-level sparsity, keep ratio 0.8 · ImageNet-1K val | Top-1 accuracy: 79.7% vs. 78.4% | Hurt |
| Use two-phase forgetting-based example removal instead of online filtering (ablation) | DeiT-S example-level sparsity, keep ratio 0.8 · ImageNet-1K val | training time reduced: 16.1% vs. 19.8% | Hurt |
| Update examples 5 times rather than 10 times (ablation) | DeiT-S example-level sparsity, keep ratio 0.8 · ImageNet-1K val | Top-1 accuracy: 78.7 -> 79.7 | Helped |
| Shuffle the removed-example pool before restoring examples (ablation) | DeiT-S example-level sparsity · ImageNet-1K val | Top-1 accuracy: 79.4% -> 79.7% | Helped |
| Select removed examples across classes rather than enforcing class balance (ablation) | DeiT-S example-level sparsity, keep ratio 0.8 · ImageNet-1K val | Top-1 accuracy: 78.4% -> 79.4% | Helped |
| Use the third-layer rather than last-layer [CLS] attention map for example scoring (ablation) | DeiT-T Tri-Level, keep ratio 0.9 · ImageNet-1K val | Top-1 accuracy: 72.4% -> 72.6% | Helped |
| Use example-level sparsity at keep ratio 0.9 (ablation) | DeiT-S · ImageNet-1K val | Top-1 accuracy: 79.8 -> 80.1 | Helped |
| Use token-level sparsity at keep ratio 0.7 (ablation) | DeiT-S · ImageNet-1K val | training time reduced: 20.6% | Helped |
Closest papers by meaning
- Chasing Sparsity in Vision Transformers: An End-to-End Exploration (NeurIPS 2021 · similarity 0.94)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.91)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.90)
- DeepCompress-ViT: Rethinking Model Compression to Enhance Efficiency of Vision Transformers at the Edge (CVPR 2025 · similarity 0.89)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.89)
- A General and Efficient Training for Transformer via Token Expansion (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.