Research map / Architectures and efficiency / Attention and transformers
Dense Vision Transformer Compression with Few Samples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DC-ViT, a few-shot ViT compression framework that drops attention modules, shrinks MLPs, and selects blocks via synthetic DeepDream-style images, producing densely spaced compressed models.
Problem
Few-shot (tiny, even unlabeled, training set) compression of Vision Transformers is unexplored, and existing few-shot block-pruning methods (PRACTISE) only yield sparse compression options.
Approach
Instead of dropping whole blocks, remove only the attention module and keep a reduced MLP (randomly selected nodes, with reused weights) so the MACs target can be hit densely; select blocks to compress using a metric loss on synthetic images generated from the pretrained model, which correlates better with post-finetuning error than training loss on the tiny set; progressively prune and finetune o
Key results
- On ViT-Base with 50 samples at 15.5G MACs, DC-ViT-B reaches 82.88% top-1 vs 71.28% for PRACTISE; at 12.7G MACs with 1000 samples, 81.60% vs 72.91% (k=3 setting, same-epoch comparison).
- On ResNet-34 with 50 samples, DC-ViT gets 72.2% top-1 at 16.5% latency reduction vs 70.3% for PRACTISE.
- Transfer: DC-ViT-B (24.8% latency reduction) achieves 99.3% on Flowers and 98.0% on CIFAR-10, comparable to original ViT-B on downstream tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace attention modules and part of the MLP with DC-ViT's dense compression | ViT-Base compressed with 500 samples · ImageNet-1k validation, 15.5 × 109 MACs | Top-1 accuracy: 73.32±0.14 -> 83.73±0.14 | Helped |
| Apply DC-ViT compression | ResNet-34 · ImageNet-1k validation | Top-1 accuracy: 70.3 -> 72.2 | Helped |
| Apply DC-ViT compression | MobileNetV2 · ImageNet-1k validation | Top-1 accuracy: 69.3 -> 69.8 | Helped |
| Use progressive compression instead of top-k compression (ablation) | DC-ViT, ViT-Base · ImageNet-1k validation, 15% and 30% MACs reduction | Top-1 accuracy: 82.84 -> 83.25; 67.95 -> 79.28; 83.05 -> 83.25; 78.50 -> 79.28 | Helped |
| Finetune only the front part of blocks rather than the entire ViT (ablation) | DC-ViT, ViT-Base · ImageNet-1k validation, 15% and 30% MACs reduction | Top-1 accuracy: 82.66 -> 83.25; 78.17 -> 79.28 | Helped |
| Compress k blocks rather than (k + 1) blocks (ablation) | DC-ViT, ViT-Base · ImageNet-1k validation, 15% and 30% MACs reduction | Top-1 accuracy: 82.84 -> 83.25; 75.07 -> 79.28 | Helped |
| Reuse pretrained MLP weights rather than re-initialize them (ablation) | DC-ViT, ViT-Base · ImageNet-1k validation, 15% and 30% MACs reduction | Top-1 accuracy: 82.60 -> 83.25; 77.29 -> 79.28 | Helped |
| Use GeLU-based weights reuse instead of the default MLP weight reuse (ablation) | DC-ViT, ViT-Base · ImageNet-1k validation, 15% and 30% MACs reduction | Top-1 accuracy: 83.07; 79.25 | No clear effect |
| Use MSE loss instead of the default metric for choosing candidate models (ablation) | DC-ViT, ViT-Base · ImageNet-1k validation, 15% and 30% MACs reduction | Top-1 accuracy: 74.02; 69.22 | Hurt |
| Perform feature mimicking on the class token only (ablation) | DC-ViT, ViT-Base · ImageNet-1k validation, 15% and 30% MACs reduction | Top-1 accuracy: 82.73; 78.33 | Hurt |
Surprising findings
- Synthetic-image metric loss tracks post-finetuning top-1 error across blocks, while training loss on the tiny training set does not correlate.
- Compressing as few blocks as possible is preferable; accuracy drops more when more blocks are compressed for the same MACs target.
Limitations
- Compressed MACs ratio cannot be too large (mostly <=30%) before accuracy drops sharply.
- Evaluated only on recognition tasks; detection and segmentation untested.
- Few-shot baselines for ViT compression are limited; other ViT pruning methods did not converge in few-shot settings.
Experimental setup
Datasets: ImageNet-1k CIFAR-10 CIFAR-100 Flowers Oxford-IIIT Pets CUB-200 Indoor67
Benchmarks: ImageNet-1k CIFAR-10 CIFAR-100 Flowers Oxford-IIIT Pets CUB-200 Indoor67
Metrics: top-1 accuracy top-5 accuracy latency (ms) MACs
Baselines: PRACTISE FSKD CD MiR S2ViTE EViT AutoFormer Mini-DeiT VTP RegNetY-16GF
Builds on: PRACTISE MiR DeepDream
Closest papers by meaning
- Unified Visual Transformer Compression (ICLR 2022 · similarity 0.89)
- COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Models (ICML 2023 · similarity 0.88)
- DeepCompress-ViT: Rethinking Model Compression to Enhance Efficiency of Vision Transformers at the Edge (CVPR 2025 · similarity 0.88)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.84)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.83)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.