Research map / Architectures and efficiency / Attention and transformers
Elastic ViTs from Pretrained Models without Retraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SnapViT, a retraining-free, label-free post-pretraining structured pruning method that combines self-supervised (DINO-loss) gradient-based local Hessian diagonal scores with cross-structure correlations learned by xNES evolution, yielding elastic ViTs for any sparsity in a single shot.
Problem
Pretrained vision foundation models come in only a few fixed sizes, so users cannot choose a model matching arbitrary compute budgets; existing pruning methods usually need retraining, labels, classification heads, or a fixed target sparsity.
Approach
Decompose the Hessian into a local diagonal term from self-supervised DINO-loss gradients and a global cross-block correlation term learned implicitly by xNES evolution strategy (its covariance approximating the inverse Hessian) using a label-free cosine-similarity fitness across multiple sparsities, then globally rank structures once for single-shot elastic pruning.
Key results
- Pruning DINO ViT-B/16 to 40% sparsity gives under 5% accuracy degradation with 1.58x inference speedup, and linear classification gains of 15.1% over SNIP Magnitude and 53.2% over LAMP.
- On supervised AugReg ViT-B/16, the label-free method improves over SNIP and FPTP by 7% and 12.3% respectively at 50% sparsity in k-NN and linear classification.
- After full fine-tuning of DeiT ViT-B/16 at 50% sparsity for 300 epochs, SnapViT reaches 82.6% ImageNet-1k accuracy, versus 83.3% for NViT, with better average k-NN (75.4 vs 73.7) and linear (75.9 vs 72.0) accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add cross-network interactions between FFN blocks and attention heads rather than modeling none (ablation) | SnapViT on DINO ViT-B/16 · 50% sparsity | Avg. k-NN accuracy: 56.6 -> 63.5; +6.9% | Helped |
| Increase genetic algorithm iterations from 50 to 500 (ablation) | SnapViT on AugReg ViT-B/16 · 50% sparsity | Avg. Linear accuracy: 42.2 -> 44.0 | Helped |
| Optimize the genetic algorithm for more target sparsities (ablation) | SnapViT · Image classification; sparsity levels | k-NN accuracy | Helped |
| Replace identity covariance initialization with CKA-score initialization (ablation) | SnapViT on SigLIPv2 ViT-B/16 · 30% sparsity, ImageNet-1k k-NN | accuracy: 35.8 -> 56.9 | Helped |
| Replace identity covariance initialization with CKA-score initialization (ablation) | SnapViT on SigLIPv2 ViT-B/16 · 30% sparsity, ImageNet-1k linear classification | accuracy: 33.9 -> 54.9 | Helped |
| Use cross-entropy gradients instead of self-supervised gradients (ablation) | SnapViT on AugReg ViT-B/16 · Image classification | accuracy | Helped |
| Optimize each target sparsity individually rather than using one-shot continuous optimization (ablation) | SnapViT · Image classification at target sparsities | accuracy | No clear effect |
| Use Merged Data rather than ImageNet-1k pruning data (ablation) | SnapViT on DINO ViT-B/16 · 60% sparsity, evaluation datasets | Avg. top-1 accuracy: +5.3% | Helped |
| Use ImageNet-1k rather than Shaders-21k pruning data (ablation) | SnapViT on DINO ViT-B/16 · 40% sparsity, evaluation datasets | Avg. top-1 accuracy: +6.4% | Helped |
Surprising findings
- Self-supervised DINO-loss gradients perform comparably to cross-entropy gradients for pruning supervised AugReg ViT-B/16, with cross-entropy only marginally better.
- Very large foundation models pretrained on massive data (SigLIPv2 ViT-G/16, DINOv3 ViT-H+/16) degrade sharply beyond 30% sparsity, unlike DeiT-III ViT-H/14 which remains stable to 50%.
Limitations
- Large-scale self-supervised foundation models (DINOv3, SigLIPv2) are harder to prune, with sharp accuracy drops beyond 30% sparsity without weight correction.
- Comparisons to LLM Surgeon are not one-to-one because it prunes independent rows and columns, and the baselines were run with reduced shots or samples.
- The covariance-as-inverse-Hessian interpretation is an approximation justified only heuristically; results have high variance at high sparsity across seeds.
Experimental setup
Datasets: ImageNet-1k FGVC Aircraft Oxford-IIT Pets DTD Textures EuroSAT CIFAR-10 CIFAR-100 Pascal VOC 2012 DiffusionDB Shaders21k
Benchmarks: ImageNet-1k FGVC Aircraft Oxford-IIT Pets DTD Textures EuroSAT CIFAR-10 CIFAR-100 Pascal VOC 2012
Metrics: top-1 accuracy (k-NN) top-1 accuracy (linear probing) mIoU GFLOPs
Baselines: LLM Surgeon LAMP FPTP SparseGPT SNIP Magnitude NViT SN-Net LPViT SAViT
Builds on: LAMP SNIP xNES SparseGPT DINO
Closest papers by meaning
- RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers (ICML 2025 · similarity 0.88)
- Preserving Deep Representations in One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework (ICLR 2025 · similarity 0.88)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.88)
- Vision Transformers as Probabilistic Expansion from Learngene (ICML 2024 · similarity 0.88)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.87)
- MDP: Multidimensional Vision Model Pruning with Latency Constraint (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.