Research map / Architectures and efficiency / Attention and transformers
Slicing Vision Transformer for Flexible Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scala slices a ViT into many width-scaled sub-networks trained in one shot, using Isolated Activation and Scale Coordination, so a single model matches separately trained DeiT variants.
Problem
Scaling down a ViT to flexible widths under dynamic resource constraints; uniform width slicing (US-Net style) works poorly on ViTs, falling well short of separate training.
Approach
Smaller ViTs are nested sub-networks of a larger ViT by embedding width. Scala always activates the smallest subnet via reverse-direction slicing (Isolated Activation), and coordinates training with progressive KD from intermediate subnets as teacher assistants, stable sampling of widths, and CE+KL noise calibration.
Key results
- On DeiT-B 300-epoch training, Scala achieves an average 1.6% improvement over SN-Net across width ratios when using the stronger teacher.
- On DeiT-S 100 epochs, Scala reaches 58.7% at r=0.25 versus 45.8% for separate training, and 76.1% at r=1.00 versus 75.0%.
- Scala trained with 13 widths reaches 75.3% at r=0.25 on DeiT-B vs 72.2% for separate training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use one-shot Scala training rather than Separate Training | DeiT-S · ImageNet-1K, r=0.25 | Top-1 accuracy: 45.8\% -> 58.7\% | Helped |
| Use one-shot Scala training rather than Separate Training | DeiT-S · ImageNet-1K, r=0.50 | Top-1 accuracy: 65.1\% -> 68.3\% | Helped |
| Use one-shot Scala training rather than Separate Training | DeiT-B · ImageNet-1K, r=0.50 | Top-1 accuracy: 79.9\% -> 79.3\% | Hurt |
| Add Isolated Activation for the smallest subnet (ablation) | Scala · ImageNet-1K, all width ratios | Top-1 accuracy | Helped |
| Add Progressive Knowledge Transfer instead of distilling from the full network through the classification token (ablation) | Scala · ImageNet-1K, especially small width ratios | Top-1 accuracy | Helped |
| Add Stable Sampling instead of sampling intermediate width ratios uniformly across (s, l) (ablation) | Scala · ImageNet-1K | Top-1 accuracy | Helped |
| Add Noise Calibration to combine cross-entropy with teacher-guidance loss (ablation) | Scala · ImageNet-1K, small width ratios | Top-1 accuracy | Helped |
| Train from scratch rather than fine-tune the pretrained DeiT-S (ablation) | Scala · ImageNet-1K, r=0.25 | Top-1 accuracy: 23.4 -> 58.7 | Helped |
| Train from scratch rather than fine-tune the pretrained DeiT-S (ablation) | Scala · ImageNet-1K, r=0.50 | Top-1 accuracy: 52.3 -> 68.3 | Helped |
| Replace Isolated Activation with activation choice (b) (ablation) | Scala · ImageNet-1K, r=0.50 | Top-1 accuracy: 68.3 -> 64.9 | Hurt |
| Replace Isolated Activation with activation choice (c) (ablation) | Scala · ImageNet-1K, r=1.00 | Top-1 accuracy: 76.1 -> 74.8 | Hurt |
| Evaluate the ViT at an unseen width ratio (ablation) | Scala · ImageNet-1K, width ratio 0.40625 | Top-1 accuracy: 68.3 -> 1.7 | Hurt |
Surprising findings
- Vanilla ViT is essentially not slimmable: ViT achieves near-zero accuracy at unseen width ratios, unlike CNNs, which interpolate well.
- Not constantly activating the smallest subnet hurts it (accuracy drop) but improves other subnets by about 1.8% on average.
Limitations
- Longer training time than conventional single-model supervised training due to multiple active subnets per iteration.
- Vanilla pre-trained ViTs are not slimmable; fine-tuning them with Scala performs worse at small widths than training from scratch.
- Decoder for segmentation was not scaled; only the backbone is slimmable.
Experimental setup
Datasets: ImageNet-1K ADE20K UCF101
Benchmarks: ImageNet-1K ADE20K UCF101 12 fine-grained classification datasets
Metrics: Top-1 accuracy GFLOPs mIoU
Baselines: AutoFormer US-Net Separate Training SN-Net MatFormer US-Net+KD ST+KD
Builds on: US-Net sandwich rule DeiT Slimmable Networks
Closest papers by meaning
- A-ViT: Adaptive Tokens for Efficient Vision Transformer (CVPR 2022 · similarity 0.90)
- Chasing Sparsity in Vision Transformers: An End-to-End Exploration (NeurIPS 2021 · similarity 0.90)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.89)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.89)
- Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.88)
- Structured Initialization for Vision Transformers (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.