Research map / Architectures and efficiency / Attention and transformers
DiffRate : Differentiable Compression Rate for Efficient Vision Transformers
Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, Ping Luo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn layer-wise compression rates with differentiable proxy instead of handcrafted schedule | token compression on off-the-shelf DeiT/MAE/LV-ViT · ImageNet-1k val | top-1 accuracy: ToMe 77.99 -> DiffRate 78.76 (ViT-S, 2.3 GFLOPs) | Improved |
| combine pruning then merging instead of pruning only (ablation) | DiffRate on ViT-B (DeiT) · ImageNet-1k val | top-1 accuracy: 80.83 -> 81.50 | Improved |
| combine pruning then merging instead of merging only (ablation) | DiffRate on ViT-B (DeiT) · ImageNet-1k val | top-1 accuracy: 81.14 -> 81.50 | Improved |
| merge-then-prune order instead of prune-then-merge (ablation) | DiffRate on ViT-B (DeiT) · ImageNet-1k val | top-1 accuracy: 81.50 -> 81.18 | Hurt |
| sort tokens by class attention instead of random rank (ablation) | DiffRate on ViT-B (DeiT) · ImageNet-1k val | top-1 accuracy: 79.72 -> 81.50 | Improved |
| use A_c times value norm instead of class attention A_c (ablation) | DiffRate sorting metric · ImageNet-1k val ViT-B (DeiT) | top-1 accuracy: 81.50 -> 81.53 | No clear effect |
| use only 1,000 images to search compression rate instead of all training data (ablation) | DiffRate · ImageNet-1k val ViT-B (DeiT) | top-1 accuracy: 81.50 -> 81.40 | No clear effect |
| extend search from 3 to 30 epochs (ablation) | DiffRate · ImageNet-1k val ViT-B (DeiT) | top-1 accuracy: 81.50 -> 81.52 | No clear effect |
| search for 3 epochs rather than 1 epoch (ablation) | DiffRate · ImageNet-1k val ViT-B (DeiT) | top-1 accuracy: 81.32 -> 81.50 | Improved |
| add latency and power constraints (DiffRate-M) to the FLOPs constraint (ablation) | DiffRate on ViT-S (DeiT) · ImageNet-1k val, Gemmini accelerator | latency, power, accuracy: latency 40.1 -> 37.6 ms, power 98 -> 90 mW, accuracy 79.83 -> 79.80 | Improved |
| fine-tune for 30 epochs after searching compression rate (ablation) | DiffRate on ViT-S (DeiT) · ImageNet-1k val | top-1 accuracy: 79.58 -> 79.83 | Improved |
| apply off-the-shelf DiffRate on ViT-L (MAE) versus ToMe trained | DiffRate · ImageNet-1k val, ViT-L (MAE) | top-1 accuracy: 85.59 (ToMe trained) vs 85.56 | No clear effect |
Later papers that compared with this method
5 later accepted papers report DiffRate as a baseline in their results tables; 5 of them report a better number on most of the results they share with it.
- ToaSt: Token Channel Selection and Structured Pruning for Efficient ViT (ICML 2026 · better on 5 of 5 shared results)
- Saliency-Driven Token Merging for Vision Transformers (CVPR 2026 · better on 2 of 2 shared results)
- Frequency-Aware Token Reduction for Efficient Vision Transformer (NeurIPS 2025 · better on 2 of 2 shared results)
- Accelerating Transformers with Spectrum-Preserving Token Merging (NeurIPS 2024 · better on 2 of 2 shared results)
- PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation (ECCV 2024 · better on 2 of 2 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers (CVPR 2025 · similarity 0.91)
- Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Acceleration (CVPR 2025 · similarity 0.89)
- Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More (ICML 2025 · similarity 0.87)
- Frequency-Aware Token Reduction for Efficient Vision Transformer (NeurIPS 2025 · similarity 0.87)
- SparseDiT: Token Sparsification for Efficient Diffusion Transformer (NeurIPS 2025 · similarity 0.87)
- PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
