Research map / Architectures and efficiency / Attention and transformers
Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decomposes a pretrained-scale ViT (DeiT-Base) into reusable linear "learngenes" weighted by Chebyshev polynomial coefficients, then recomposes them into diverse-depth ViTs that initialize or train for downstream classification without retraining from scratch.
Problem
Standard ViTs have fixed architectures, but deployment devices have varying compute and memory budgets. Producing many differently sized models via compression or distillation requires a separate training run per size.
Approach
Inspired by polynomial decomposition in calculus, each ViT layer's parameters are expressed as a linear combination of shared learngenes with layer-dependent, pre-defined first-kind Chebyshev coefficients. Learngenes are trained incrementally (one group at a time, earlier ones frozen), and any number of layers and learngenes can then be recomposed without extra training.
Key results
- Decomposing DeiT-Base into 12 learngenes trained incrementally reaches 83.70% ImageNet-1K accuracy, comparable to DeiT-Base, and outperforms SN-Net-stitched models at similar parameter counts.
- A 12-layer ViT recomposed from one learngene with 'w/ const' training (7.8M params) reaches 72.3% on INAT-style comparison in Figure 6(a), exceeding Efficient-ViT (8.8M) by 6.3%.
- The method claims about 80x training-cost reduction for generating 144 ViT variants (43.2K vs. about 800 epochs total), though this is a computed estimate rather than a measured result.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 12 learngenes to recompose and fine-tune ViTs | Learngene decomposition-recomposition · CIFAR-10 | accuracy: 98.92 -> 98.97 | Helped |
| Use 12 learngenes to recompose and fine-tune ViTs | Learngene decomposition-recomposition · CIFAR-100 | accuracy: 90.29 -> 91.05 | Helped |
| Use 12 learngenes to recompose and fine-tune ViTs | Learngene decomposition-recomposition · Tiny-ImageNet | accuracy: 87.5 -> 89.16 | Helped |
| Use 12 learngenes to recompose and fine-tune ViTs | Learngene decomposition-recomposition · INAT-2019 | accuracy: 77.43 -> 77.89 | Helped |
| Use 12 learngenes to recompose and fine-tune ViTs | Learngene decomposition-recomposition · Food-101 | accuracy: 91.44 -> 92.02 | Helped |
| Use 12 learngenes to recompose and fine-tune ViTs | Learngene decomposition-recomposition · Stanford Cars | accuracy: 91.97 -> 92.68 | Helped |
| Use 12 learngenes and train with constraints | Learngene decomposition-recomposition · Oxford Flowers-102 | accuracy: 92.54 -> 95.8 | Helped |
| Use 12 learngenes and train with constraints | Learngene decomposition-recomposition · Oxford-IIIT Pets | accuracy: 93.71 -> 95.59 | Helped |
| Use 12 learngenes to recompose and fine-tune ViTs | Learngene decomposition-recomposition · DTD | accuracy: 72.87 -> 74.73 | Helped |
| Use first-kind Chebyshev coefficients instead of Taylor coefficients (ablation) | Decomposition coefficient selection · ImageNet-1K | accuracy: 74.80% -> 77.73% | Helped |
| Use first-kind Chebyshev coefficients instead of Fourier coefficients (ablation) | Decomposition coefficient selection · ImageNet-1K | accuracy: 77.09% -> 77.73% | Helped |
| Train learngenes incrementally rather than simultaneously (ablation) | DeiT-Base decomposition · ImageNet-1K | accuracy: 81.85 -> 83.70 | Helped |
Surprising findings
- Incremental learngene training (ICT) surpasses simultaneous training (SCT) at the same learngene counts, as the number of learngenes grows.
- The 'w/o const' recomposition performs worse than baselines with few layers (e.g., a 4-layer model is over 9% below DeiT-Small), though it becomes competitive with more layers.
Limitations
- Lacks theoretical grounding; the linear-decomposition assumption rests on an empirical observation from PCA of layer parameters.
- Results are limited to ViT classification on nine downstream datasets with a single training seed and no error bars.
- Recomposed models with few layers and 'w/o const' training underperform the pre-train-and-fine-tune baselines.
Experimental setup
Datasets: ImageNet-1K CIFAR-10 CIFAR-100 Tiny-ImageNet iNaturalist-2019 Food-101 Oxford Flowers-102 Stanford Cars Oxford-IIIT Pets DTD
Benchmarks: ImageNet-1K CIFAR-10 CIFAR-100 Tiny-ImageNet iNaturalist-2019 Food-101 Oxford Flowers-102 Stanford Cars Oxford-IIIT Pets DTD
Metrics: top-1 accuracy parameter count memory size
Baselines: DeiT-Tiny DeiT-Small DeiT-Base (pre-train and fine-tune) Mini-DeiT-B (MiniViT) Efficient-ViT SN-Net
Builds on: Learngene DeiT MiniViT SN-Net
Closest papers by meaning
- Vision Transformers as Probabilistic Expansion from Learngene (ICML 2024 · similarity 0.89)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.89)
- DeepCompress-ViT: Rethinking Model Compression to Enhance Efficiency of Vision Transformers at the Edge (CVPR 2025 · similarity 0.89)
- Split Adaptation for Pre-trained Vision Transformers (CVPR 2025 · similarity 0.88)
- Initializing Variable-sized Vision Transformers from Learngene with Learnable Transformation (NeurIPS 2024 · similarity 0.88)
- Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.