Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Unlocking Pre-trained Weights: Parameter Inheritance for Zero-Shot Initialization
Jiaze Xu, Shiyu Xia, Jiaqi Lv, Xin Geng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| inherit pre-trained ViT-Large weights through learned projection (PITH) instead of soft-label supervision (TAL) | ViT-Base hypernetwork initialization · ImageNet-1K zero-shot | top-1 accuracy (%): 46.81 -> 53.35 | Improved |
| add pre-trained parameter projection (ablation) | PITH · ImageNet-1K untrained average | average accuracy (%): 46.90 -> 49.02 | Improved |
| add progressive dimension expansion (ablation) | PITH · ImageNet-1K untrained average | average accuracy (%): 46.43 -> 49.02 | Improved |
| share MLP layers between weight and projection pathways (ablation) | PITH · ImageNet-1K untrained average | average accuracy (%): 45.73 -> 46.08 vs LoGAH, params 21.41M -> 24.40M | Improved |
| use full dual-pathway decoder instead of shared pathway (ablation) | PITH · ImageNet-1K untrained average | average accuracy (%): 46.08 -> 46.43 with params 24.40M -> 41.59M | Improved |
| PITH instead of TAL trained longer (300 epochs) | hypernetwork training budget · Decathlon average accuracy | average accuracy (%): 47.54 -> 51.03 | Improved |
| PITH initialization instead of TAL | ViT-Tiny · Decathlon trained 100 epochs | average accuracy (%): 58.98 -> 61.65 | Improved |
| PITH initialization instead of TAL | ViT-Small · Decathlon trained 100 epochs | average accuracy (%): 58.39 -> 60.73 | Improved |
| PITH initialization instead of TAL | ViT-Tiny 12 layers · Decathlon untrained, DPed | accuracy (%): 93.11 -> 90.56 | Hurt |
| PITH initialization instead of TAL | ViT-Small · FER2013 unseen task, 20 epochs | accuracy (%): 53.50 -> 52.55 | Hurt |
| PITH initialization instead of RandInit | ViT-Tiny · HAM10000 unseen task, 100 epochs | accuracy (%): 97.34 -> 97.71 | No clear effect |
| PITH instead of TAL | training cost · 200 epochs on RTX Pro 6000 | training time (h): 92.37 -> 95.43 | No clear effect |
Closest papers by meaning
- Learngene Tells You How to Customize: Task-Aware Parameter Initialization at Flexible Scales (ICML 2025 · similarity 0.89)
- Unlocking the Potential of Black-box Pre-trained GNNs for Graph Few-shot Learning (AAAI 2025 · similarity 0.87)
- Principled Architecture-aware Scaling of Hyperparameters (ICLR 2024 · similarity 0.86)
- IDInit: A Universal and Stable Initialization Method for Neural Network Training (ICLR 2025 · similarity 0.85)
- If You Want to Be Robust, Be Wary of Initialization (NeurIPS 2024 · similarity 0.85)
- LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
