Research map / Architectures and efficiency / Attention and transformers
Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OFB, a single-stage vision transformer compression method that jointly learns importance and sparsity scores via a bi-mask weight-sharing scheme, an adaptive one-hot loss, and progressive masked image modeling.
Problem
Vision Transformer compression (TAS and TP) typically uses a two-stage importance-then-sparsity search, causing a gap between score distributions and high search costs.
Approach
Entangles importance scores (learned soft masks with l1 regularization) and differentiable sparsity scores (DARTS-style softmax over step-wise architecture parameters) into a single bi-mask that is optimized jointly, with an adaptive one-hot regularizer (entropy and variance alignment) that converts continuous scores to binary pruning decisions, plus a progressive MIM regularizer during search.
Key results
- On ImageNet-1K, OFB compresses DeiT-S to 4.4M params / 0.9B FLOPs with 75.0% Top-1 at about 1 GPU search day, versus ViTAS-B 72.4% at 32 GPU days.
- OFB compresses DeiT-B to 43.9M params / 8.7B FLOPs with 81.7% Top-1 (−0.1%) in 4 GPU days, outperforming WDPruning by 0.9% Top-1 at similar FLOPs.
- Ablation shows the bi-mask weight-sharing strategy gives +1.7% Top-1 and PMIM gives +3.6% Top-1 before retraining on DeiT-S.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use bi-mask weight sharing instead of ordinal sharing (ablation) | OFB on DeiT-S · ImageNet-1K, 1B FLOPs search ablation | Top-1 accuracy: 75.8 -> 76.1 | Helped |
| add progressive masked image modeling to bi-mask (ablation) | OFB on DeiT-S · ImageNet-1K, search without retraining | Top-1 accuracy: 69.2 -> 72.8 | Helped |
| use progressive masking instead of constant masking (ablation) | OFB on DeiT-S · ImageNet-1K, 1B FLOPs | Top-1 accuracy: 75.8 -> 76.1 | Helped |
| remove masked image modeling (ablation) | OFB on DeiT-S · ImageNet-1K, 1B FLOPs | Top-1 accuracy: 76.1 -> 75.4 | Hurt |
| use variance regularization Ψ(α) only instead of the full regularization (ablation) | OFB on DeiT-S · ImageNet-1K, regularization ablation | Top-1 accuracy: 76.1 -> 74.1 | Hurt |
| use L1 regularization only instead of the full regularization (ablation) | OFB on DeiT-S · ImageNet-1K, regularization ablation | Top-1 accuracy: 76.1 -> 74.8 | Hurt |
| combine entropy and variance regularization without L1 (ablation) | OFB on DeiT-S · ImageNet-1K, regularization ablation | Top-1 accuracy: 76.1 -> 74.3 | Hurt |
| combine entropy and L1 regularization without variance regularization (ablation) | OFB on DeiT-S · ImageNet-1K, regularization ablation | Top-1 accuracy: 76.1 -> 76.3 | Helped |
| combine variance and L1 regularization without entropy regularization (ablation) | OFB on DeiT-S · ImageNet-1K, regularization ablation | Top-1 accuracy: 76.1 -> 75.6 | Hurt |
| remove variance regularization from the full regularization (ablation) | OFB on DeiT-S · ImageNet-1K, regularization ablation | Top-1 accuracy: 76.1 -> 76.3 | Helped |
| compress DeiT-S using OFB instead of the strongest listed comparator at 0.9B FLOPs | OFB on DeiT-S · ImageNet-1K, 0.9B FLOPs | Top-1 accuracy: 75.3 -> 75.0 | Hurt |
| compress DeiT-S using OFB instead of ViT-Slim at 1.7B FLOPs | OFB on DeiT-S · ImageNet-1K, 1.7B FLOPs | Top-1 accuracy: 77.9 -> 78.0 | Helped |
Surprising findings
- Retraining after search gives little benefit for DeiT-B-8.7B, which already reaches retrained-level accuracy after search.
Limitations
- Search and pruning are tied to specific search spaces (Q-K-V ratio, MLP ratio, head number, patch embedding) defined per architecture.
- Evaluation is mainly on ImageNet-1K with DeiT and Swin; generalization to other architectures is limited to Swin-Ti.
Experimental setup
Datasets: ImageNet-1K CIFAR-10 CIFAR-100 Cityscapes
Benchmarks: ImageNet-1K CIFAR-10 CIFAR-100 Cityscapes
Metrics: Top-1 accuracy Top-5 accuracy FLOPs parameters GPU search days throughput latency mIoU
Baselines: AutoFormer ViTAS TF-TAS ViT-Slim S2ViTE WDPruning SSP DynamicViT VTP GLiT
Builds on: DARTS ViT-Slim SimMIM SPOS AutoFormer
Closest papers by meaning
- Unified Visual Transformer Compression (ICLR 2022 · similarity 0.88)
- VTC-LFC: Vision Transformer Compression with Low-Frequency Components (NeurIPS 2022 · similarity 0.87)
- Chasing Sparsity in Vision Transformers: An End-to-End Exploration (NeurIPS 2021 · similarity 0.86)
- Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training (AAAI 2023 · similarity 0.86)
- Benchmarking and Enhancing VLM for Compressed Image Understanding (ICML 2026 · similarity 0.85)
- Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.