Research map / Architectures and efficiency / RNNs and normalization
Building Variable-Sized Models via Learngene Pool
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills one pre-trained DeiT-Base into small auxiliary models whose blocks form a learngene pool, then stitches blocks with learnable transformation-initialized stitch layers to build variable-sized ImageNet classifiers.
Problem
SN-Net stitches multiple independently pre-trained anchors, which costs much storage, cannot produce very small models under low resource constraints, and initializes stitch layers with an unlearnable least-squares method that limits performance.
Approach
Distill a single large model (DeiT-Base) via prediction, block, and attention distillation with learnable transformation matrices into small auxiliary models; treat their blocks as learngene instances forming a pool; initialize stitch layers from the learned distillation transformation matrices instead of least squares.
Key results
- Learngene Pool (12) saves about 59.6% storage versus SN-Net's 118.4M (47.83M) and reaches 75.05% at 44.04M params vs SN-Net's 67.98%.
- Learngene Pool (18) reaches 77.42% at 65.30M params vs SN-Net's 69.11%, and builds models as small as 3.05M/4.38M params where SN-Net cannot go below 5.7M.
- Transformation-matrix initialization of stitch layers outperforms least-squares initialization across built models (e.g., 57.00 vs 56.95 at 3.05M, 70.00 vs 69.76 at 17.02M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Learngene Pool (12) rather than SN-Net at matched storage cost | DeiT-based variable-sized models · ImageNet-1K val, 44.04M params | accuracy: 67.98 -> 75.05 | Helped |
| use Learngene Pool (18) rather than SN-Net at matched storage cost | DeiT-based variable-sized models · ImageNet-1K val, 65.30M params | accuracy: 69.11 -> 77.42 | Helped |
| increase the pool from 12 to 18 instances | Learngene Pool · ImageNet-1K val, parameter-size ranges | accuracy | Helped |
| distill three ancestry-model blocks instead of only the last block (ablation) | auxiliary models with lower output dimensions · ImageNet-1K | accuracy: 44.27 -> 53.82 (6 blocks); 54.98 -> 60.44 (9 blocks) | Helped |
| distill only one block rather than no blocks (ablation) | auxiliary models with lower output dimensions · ImageNet-1K | accuracy: 53.49 -> 44.27 (6 blocks); 58.96 -> 54.98 (9 blocks) | Hurt |
| distill one block rather than no blocks (ablation) | auxiliary models with matching output dimensions · ImageNet-1K | accuracy: 67.20 -> 78.83 (6 blocks); 68.46 -> 80.28 (9 blocks) | Helped |
| distill three blocks rather than one block (ablation) | auxiliary models with matching output dimensions · ImageNet-1K | accuracy: 78.83 -> 70.57 (6 blocks); 80.28 -> 75.38 (9 blocks) | Hurt |
| initialize stitch layers with block-based transformation matrices rather than least squares (ablation) | Learngene Pool descendant models · ImageNet-1K val | accuracy: 56.95 -> 57.00; 62.64 -> 63.77; 60.73 -> 61.63; 65.07 -> 66.95 | Helped |
| omit distillation during fine-tuning with transformation-matrix initialization (ablation) | Learngene Pool · ImageNet-1K val | accuracy | No clear effect |
| omit distillation during fine-tuning with least-squares initialization (ablation) | Learngene Pool · ImageNet-1K val | accuracy | Hurt |
Surprising findings
- Distilling only one block (last block) for lower-dimensional auxiliary models performs worse than training from scratch.
Limitations
- Evaluation limited to ImageNet-1K classification with DeiT-based architecture.
- Comparison with SN-Net is at matched storage budgets, not one-to-one model sizes, making direct comparison harder.
- Distillation with lower-dimensional auxiliary models needs multiple blocks and extra design choices (block selection, dense distillation levels).
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet-1K
Metrics: top-1 accuracy parameters (M) FLOPs (G)
Baselines: SN-Net DeiT-Tiny training from scratch
Builds on: Stitchable Neural Networks Learngene TinyBERT DeiT
Closest papers by meaning
- Efficient Stitchable Task Adaptation (CVPR 2024 · similarity 0.88)
- Initializing Variable-sized Vision Transformers from Learngene with Learnable Transformation (NeurIPS 2024 · similarity 0.85)
- Cluster-Learngene: Inheriting Adaptive Clusters for Vision Transformers (NeurIPS 2024 · similarity 0.85)
- Learngene Tells You How to Customize: Task-Aware Parameter Initialization at Flexible Scales (ICML 2025 · similarity 0.83)
- Inheriting Generalized Learngene for Efficient Knowledge Transfer across Multiple Tasks (AAAI 2025 · similarity 0.83)
- MotherNet: Fast Training and Inference via Hyper-Network Transformers (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.