Research map / Architectures and efficiency / Pruning and efficient tuning
Initializing Variable-sized Vision Transformers from Learngene with Learnable Transformation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LeTs, which trains a compact learngene plus learnable width and depth transformation matrices via distillation from an ancestry model, then selects matrix slices to initialize variable-sized ViTs that are fine-tuned on downstream tasks.
Problem
Initializing variable-sized vision transformers under diverse resource constraints; prior Learngene methods use manually designed, depth-only transformations of learngene without learnable components.
Approach
Learn compact learngene layers together with learnable width (in/out) and depth transformation matrices by distilling an auxiliary model from an ancestry model, then select rows/entries of these trained matrices (continuous, magnitude-wise, sequential selection) to transform learngene into Des-Nets of any width and depth.
Key results
- Des-Nets initialized via LeTs outperform 100-epoch from-scratch training after only 1 epoch of tuning on ImageNet-1K, reducing total training cost about 7x.
- Without any tuning, LeTs reaches 80.1% on Des-H12-L12 versus 76.6% for TLEG (direct evaluation on ImageNet-1K, +3.5%).
- On CIFAR-100 with Des-H12-L12, LeTs (90.0%) beats Pre-Fin (88.5%) while reducing initialization parameter storage about 20x (758M vs 37.7M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| initialize with LeTs rather than Scratch | DeiT Des-H12-L12 · ImageNet-1K | Top-1 accuracy (%): 79.6 -> 80.1 | Helped |
| initialize with LeTs rather than SWS | DeiT Des-H6-L12 · CIFAR-100 | Top-1 accuracy (%): 87.4 -> 89.2 | Helped |
| initialize with LeTs rather than SWS | DeiT Des-H12-L12 · CIFAR-10 | Top-1 accuracy (%): 98.9 -> 98.9 | No clear effect |
| initialize with LeTs rather than SWS | DeiT Des-H6-L12 · Food-101 | Top-1 accuracy (%): 89.2 -> 90.8 | Helped |
| initialize with LeTs rather than SWS | DeiT Des-H6-L12 · Cars-196 | Top-1 accuracy (%): 84.5 -> 87.1 | Helped |
| initialize with LeTs rather than Pre-Fin | DeiT Des-H6-L12 · ADE20K | mIoU (%): 36.8 -> 40.2 | Helped |
| select bottom-n rather than top-n rows by L2-norm (ablation) | LeTs, DeiT Des-H8-L12 · ImageNet-1K | Top-1 accuracy (%): 79.3 -> 78.8 | Hurt |
| select bottom-n rather than top-n rows by L1-norm (ablation) | LeTs, DeiT Des-H8-L12 · ImageNet-1K | Top-1 accuracy (%): 79.4 -> 78.7 | Hurt |
| remove MSA from initialization (ablation) | LeTs, DeiT Des-H8-L12 · ImageNet-1K | Top-1 accuracy (%) | Hurt |
| remove MLP from initialization (ablation) | LeTs, DeiT Des-H8-L12 · ImageNet-1K | Top-1 accuracy (%) | Hurt |
| remove layer normalization from initialization (ablation) | LeTs, DeiT Des-H8-L12 · ImageNet-1K | Top-1 accuracy (%) | Hurt |
| remove patch embedding from initialization (ablation) | LeTs, DeiT Des-H8-L12 · ImageNet-1K | Top-1 accuracy (%) | Hurt |
Surprising findings
- Initializing from LeTs without any tuning on ImageNet-1K slightly exceeds the 100-epoch from-scratch baseline for some sizes (e.g., Des-H12-L12 80.1% vs. 79.6% Scratch, as reported in the table).
Limitations
- Hyperparameters for Aux-Net construction and training and fine-tuning were not carefully tuned (initialization order etc.).
- Only prediction-based distillation is used to train learngene and transformation matrices, which may limit quality.
- Evaluated only on ViT image classification and segmentation; language tasks left for future work.
Experimental setup
Datasets: ImageNet-1K CIFAR-10 CIFAR-100 Food-101 Cars-196 ADE20K Pascal Context Cityscapes
Benchmarks: ImageNet-1K CIFAR-10 CIFAR-100 Food-101 Cars-196 ADE20K Pascal Context Cityscapes
Metrics: Top-1 accuracy mIoU Params (M) FLOPs (G)
Baselines: Scratch TLEG SWS Grad-LG IMwLM Pre-Fin MatFormer LiGO
Builds on: Learngene LiGO DeiT Net2Net TLEG
Closest papers by meaning
- Vision Transformers as Probabilistic Expansion from Learngene (ICML 2024 · similarity 0.91)
- Transformer as Linear Expansion of Learngene (AAAI 2024 · similarity 0.90)
- Cluster-Learngene: Inheriting Adaptive Clusters for Vision Transformers (NeurIPS 2024 · similarity 0.89)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.88)
- Inheriting Generalized Learngene for Efficient Knowledge Transfer across Multiple Tasks (AAAI 2025 · similarity 0.85)
- Building Variable-Sized Models via Learngene Pool (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.