Research map / Architectures and efficiency / Pruning and efficient tuning
WAVE: Weight Templates for Adaptive Initialization of Variable-sized Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WAVE, which initializes variable-sized ViTs (depth and width) from shared, distilled weight templates combined via Kronecker-product weight scalers trained on minimal data.
Problem
Pre-trained models come in fixed sizes, so initializing target ViTs of arbitrary depth or width (incompatible with pre-trained sizes) requires costly re-pretraining or lossy weight transfer.
Approach
Reformulates variable-sized initialization as a multi-task problem: size-agnostic weight templates (learngenes) distilled from a pre-trained model via an auxiliary model, and per-size lightweight weight scalers that combine the frozen templates via Kronecker products.
Key results
- WAVE reaches 79.2% top-1 on a 12-layer DeiT-B-width model after 10 epochs, versus 77.1% for TLEG and 56.7% for He-Init on the same size grid (Table 1).
- Models initialized by WAVE trained for 10 epochs outperform models directly pre-trained for 150 epochs, with 15x computational savings for 15 variable-sized models (Figure 3).
- On width expansion to W1536 from L6, WAVE achieves 78.3% versus 73.7% for LiGO (Table 2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Kronecker-product template concatenation instead of linear combination (ablation) | WAVE · ImageNet-1K, L4 W768 | Top-1 accuracy: 71.6 -> 74.5 | Helped |
| use Kronecker-product initialization instead of weight transformation (ablation) | WAVE · ImageNet-1K, L6 W1536 | Top-1 accuracy: 73.7 -> 78.3 | Helped |
| add weight templates for attention (ablation) | WAVE · ImageNet-1K, DeiT-Ti | Top-1 accuracy: +9.6 | Helped |
| add weight templates for projection (ablation) | WAVE · ImageNet-1K, DeiT-Ti | Top-1 accuracy: +2.1 | Helped |
| add weight templates for fully connected layers (ablation) | WAVE · ImageNet-1K, DeiT-Ti | Top-1 accuracy: +6.4 | Helped |
| add weight templates for normalization (ablation) | WAVE · ImageNet-1K, DeiT-Ti | Top-1 accuracy: +0.1 | Helped |
| reduce the number of weight templates (ablation) | WAVE · ImageNet-1K, L6 W192 | Top-1 accuracy: 63.2 -> 57.5 | Hurt |
| reduce weight-template shape to 96² (ablation) | WAVE · ImageNet-1K, L6 W192 | Top-1 accuracy: 63.2 -> 60.3 | Hurt |
| use weight templates to initialize normalization (ablation) | WAVE · ImageNet-1K, DeiT-Ti | Top-1 accuracy: 63.1 -> 63.2 | No clear effect |
| initialize for variable-sized models with WAVE rather than direct pre-training | WAVE · 15 ImageNet-1K downstream models | training epochs to outperform direct pre-training: 10 epochs vs 150 epochs | Helped |
Surprising findings
- Kronecker-based scalers need only about 0.0025M trainable parameters vs 136.84M for weight transformation (LiGO) in the widest setting, yet give higher accuracy.
Limitations
- Template construction requires a 150-epoch distillation step on ImageNet-1K from a pre-trained auxiliary model.
- Reducing template count or size degrades accuracy, so template design involves a trade-off between flexibility and performance.
Experimental setup
Datasets: ImageNet-1K Oxford Flowers CUB-200-2011 Stanford Cars CIFAR-10 CIFAR-100 Food-101 iNaturalist-2019
Benchmarks: ImageNet-1K Oxford Flowers CUB-200-2011 Stanford Cars CIFAR-10 CIFAR-100 Food-101 iNaturalist-2019
Metrics: Top-1 accuracy parameters transferred FLOPs
Baselines: He-Init Mimetic Init GHN-3 Weight Selection LiGO Share Init Heur-LG Auto-LG TLEG
Builds on: TLEG Learngene LiGO Mimetic initialization Layer-Wise Scaling Kernels
Closest papers by meaning
- Initializing Models with Larger Ones (ICLR 2024 · similarity 0.87)
- The Primacy of Magnitude in Low-Rank Adaptation (NeurIPS 2025 · similarity 0.85)
- Learngene Tells You How to Customize: Task-Aware Parameter Initialization at Flexible Scales (ICML 2025 · similarity 0.85)
- RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks (ICLR 2025 · similarity 0.84)
- LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail) (ICML 2025 · similarity 0.84)
- Cluster-Learngene: Inheriting Adaptive Clusters for Vision Transformers (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.