Research map / Architectures and efficiency / Attention and transformers
Unified Visual Transformer Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly optimize within-block pruning, block skipping, and knowledge distillation under a FLOPs budget | DeiT-Tiny · ImageNet | top-1 accuracy: 66.84 -> 71.3 | Helped |
| add within-block pruning to skip manipulation and knowledge distillation (ablation) | UVC on DeiT-Tiny · ImageNet | top-1 accuracy: 68.72 -> 71.3 | Helped |
| add skip manipulation to within-block pruning and knowledge distillation (ablation) | UVC on DeiT-Tiny · ImageNet | top-1 accuracy: 70.52 -> 71.3 | Helped |
| add knowledge distillation (ablation) | UVC on DeiT-Tiny · ImageNet | top-1 accuracy: 69.34 -> 71.3 | Helped |
| apply skip configuration before pruning instead of jointly optimizing both (ablation) | DeiT-Tiny compression · ImageNet | top-1 accuracy: 71.3 -> 66.84 | Hurt |
| apply pruning before skip configuration instead of jointly optimizing both (ablation) | DeiT-Tiny compression · ImageNet | top-1 accuracy: 71.3 -> 69.68 | Hurt |
| apply pruning before skip configuration at the matched compression setting instead of jointly optimizing both (ablation) | DeiT-Tiny compression · ImageNet | top-1 accuracy: 71.3 -> 70.02 | Hurt |
| compress DeiT-Tiny with UVC to 49.23% FLOPs remained | DeiT-Tiny · ImageNet | top-1 accuracy: 72.2 -> 71.3 | Hurt |
| compress DeiT-Small with UVC to 50.41% FLOPs remained | DeiT-Small · ImageNet | top-1 accuracy: 79.8 -> 78.82 | Hurt |
| compress DeiT-Base with UVC to 45.50% FLOPs remained | DeiT-Base · ImageNet | top-1 accuracy: 81.8 -> 80.57 | Hurt |
| compress T2T-ViT-14 with UVC to 60.4% FLOPs remained | T2T-ViT-14 · ImageNet | top-1 accuracy: 81.5 -> 80.4 | Hurt |
Closest papers by meaning
- Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training (AAAI 2023 · similarity 0.93)
- VTC-LFC: Vision Transformer Compression with Low-Frequency Components (NeurIPS 2022 · similarity 0.92)
- DeepCompress-ViT: Rethinking Model Compression to Enhance Efficiency of Vision Transformers at the Edge (CVPR 2025 · similarity 0.90)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.89)
- Dense Vision Transformer Compression with Few Samples (CVPR 2024 · similarity 0.89)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.