Research map / Architectures and efficiency / Pruning and efficient tuning
Learngene Tells You How to Customize: Task-Aware Parameter Initialization at Flexible Scales
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Task-Aware Learngene (TAL), a hypernetwork-based parameter initializer that is pretrained under a ViT-Base ancestry model's guidance and then multi-task tuned on Decathlon tasks to predict initial weights for flexible-scale ViT and GPT-2 models, including unseen tasks.
Problem
Graph HyperNetwork initialization struggles with large models, performs poorly on small datasets, and requires retraining the GHN for every new task, adding computation and storage overhead.
Approach
Combine the Learngene idea (condensing knowledge from a well-trained ancestry model) with a graph-hypernetwork encoder-decoder: the encoder (learngene) is distilled from the ancestry model via KL feature matching and conditioned on task embeddings through a task hypernet producing FiLM-style scale and bias, so one shared model predicts task-aware parameters across scales.
Key results
- Untrained ViT-Small initialized with TAL+ outperforms LoGAH initialization by 22.44% on ImageNet-1K.
- Models initialized with TAL outperform GHN-based initialization by an average of 24.39% accuracy across Decathlon datasets (as stated in abstract/introduction).
- TAL reduces training time to 36.19 h versus 46.55 h for LoGAH and 58.81 h for GHN-3 on RTX 4090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize ViT-Small with TAL+ instead of LoGAH | ViT-Small, 12 layers · ImageNet-1K, untrained | accuracy: 16.78 -> 39.22 | Helped |
| Initialize ViT-Small with TAL instead of LoGAH | ViT-Small, 12 layers · ImageNet-1K, after 75 epochs | accuracy: 65.41 -> 66.61 | Helped |
| Use the enhanced ViTs+-1K architecture library | TAL · ImageNet-1K, ViT-Base untrained | accuracy: 0.10 -> 38.72 | Helped |
| Replace LoGAH initialization with TAL+ initialization | ViT-Tiny, 3 layers · Decathlon C100, untrained | accuracy: 29.16 -> 50.80 | Helped |
| Replace LoGAH initialization with TAL+ initialization | ViT-Tiny, 3 layers · Decathlon SVHN, untrained | accuracy: 17.32 -> 87.37 | Helped |
| Replace LoGAH initialization with TAL+ initialization | ViT-Small, 3 layers · Decathlon DTD, untrained | accuracy: 9.36 -> 33.14 | Helped |
| Replace LoGAH initialization with TAL+ initialization | ViT-Small, 12 layers · Decathlon UCF, trained | accuracy: 33.86 -> 52.77 | Helped |
| Tune the shared TAL model on multiple tasks instead of tuning it separately per task (ablation) | TAL · Decathlon, 12-TINY | accuracy: 1.95 -> 19.8; 5.99 -> 54.89; 90.00 -> 98.61; 6.65 -> 30.21; 82.08 -> 99.57; 0.03 -> 63.57; 27.22 -> 90.38; 13.78 -> 48.1 | Helped |
| Add task information to learngene (ablation) | TAL · Decathlon | average accuracy: 40.71 -> 53.37 | Helped |
| Add ancestry-model guidance (ablation) | TAL · Decathlon | average accuracy: 46.80 -> 53.37 | Helped |
| Initialize with TAL rather than LoGAH | GPT-2 models · MRPC | accuracy: 55.88/ 68.75 -> 62.99/ 76.70; 59.07/ 72.12 -> 61.52/ 73.70; 60.78/ 74.19 -> 58.09/ 68.97; 56.13/ 68.98 -> 52.70/ 60.53 | Mixed |
| Initialize with TAL rather than LoGAH | GPT-2 models · IMDB | accuracy: 52.06 -> 63.21; 53.58 -> 62.76; 57.76 -> 62.09; 58.07 -> 61.49 | Helped |
Surprising findings
- Untrained models initialized with TAL/TAL+ outperform models trained from other initialization methods, per the conclusion.
Limitations
- TAL+ requires substantially longer training time (70.53 h) than prior methods.
- Experiments are limited to ViT and GPT-2 architectures and a fixed set of Decathlon tasks.
Experimental setup
Datasets: ImageNet-1K Visual Decathlon ViTs-1K ViTs+-1K GPTs-1K MRPC CoLA RTE IMDB Fashion MNIST FER2013 HAM10000
Benchmarks: ImageNet-1K Visual Decathlon MRPC CoLA RTE IMDB Fashion MNIST FER2013 HAM10000
Metrics: top-1 accuracy training time (hours)
Baselines: GHN-3 LoGAH RandInit TAL-ST TAL(w/o T.I.) TAL(w/o ans-net)
Builds on: Graph HyperNetwork (GHN-2/GHN-3) LoGAH Learngene Vanilla-LG temperature-based task sampling
Closest papers by meaning
- WAVE: Weight Templates for Adaptive Initialization of Variable-sized Models (CVPR 2025 · similarity 0.85)
- From Instance Training to Instruction Learning: Task Adapters Generation from Instructions (NeurIPS 2024 · similarity 0.84)
- Cluster-Learngene: Inheriting Adaptive Clusters for Vision Transformers (NeurIPS 2024 · similarity 0.83)
- PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.83)
- Efficient Model Editing with Task-Localized Sparse Fine-tuning (ICLR 2025 · similarity 0.83)
- AdapterGNN: Parameter-Efficient Fine-Tuning Improves Generalization in GNNs (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.