Research map / Architectures and efficiency / Pruning and efficient tuning
Initializing Models with Larger Ones
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes weight selection, which initializes a smaller model by uniformly selecting a consistent subset of weights from a larger pretrained model of the same family, and shows gains on image classification and compatibility with knowledge distillation.
Problem
Weight initialization for small models is usually random (e.g., truncated normal, Xavier, Kaiming), even though large pretrained models of the same family are available; the paper asks how to use them to initialize smaller models at no extra cost.
Approach
Initialize a smaller model by selecting a subset of weights from a pretrained larger model in three steps (layer selection, component mapping, element selection), using uniform element selection with consistent indices across all weight tensors so that residual-stream neurons are preserved.
Key results
- Weight selection improves ViT-T test accuracy on all 9 datasets, e.g., CIFAR-100 from 72.3 to 81.4 and Pets from 25.0 to 68.6, and ConvNeXt-F gains up to 14.4 points on Pets.
- Weight selection reaches the same CIFAR-100 accuracy with about 1/3 of the training epochs of random initialization, and is 6.12x faster than ImageNet-1K pretraining followed by fine-tuning.
- Combining weight selection with logit-based distillation reaches 76.0% on ImageNet-1K and feature-based distillation reaches 83.9% on CIFAR-100 for ViT-T.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize with weight selection from a pretrained larger model | ViT-T · CIFAR-100 | test accuracy: 72.3 -> 81.4 | Helped |
| Initialize with weight selection from a pretrained larger model | ConvNeXt-F · CIFAR-100 | test accuracy: 81.4 -> 84.4 | Helped |
| Use consistent random indices across weight tensors (ablation) | Weight selection on ViT-T · CIFAR-100 | test accuracy: 77.4 -> 81.7 | Helped |
| Use consistent random indices across weight tensors (ablation) | Weight selection on ConvNeXt-F · CIFAR-100 | test accuracy: 82.8 -> 83.9 | Helped |
| Combine logit-based distillation with weight selection | ViT-T · ImageNet-1K | test accuracy: 74.8 -> 76.0 | Helped |
| Combine feature-based distillation with weight selection | ViT-T · CIFAR-100 | test accuracy: 78.4 -> 83.9 | Helped |
| Select first-N rather than evenly spaced teacher layers (ablation) | Weight selection on ViT-A · CIFAR-100 | test accuracy: 76.7 -> 77.6 | Helped |
| Select first-N rather than evenly spaced teacher layers (ablation) | Weight selection on ConvNeXt-F · CIFAR-100 | test accuracy: 83.2 -> 84.4 | Helped |
| Use L1 pruning instead of weight selection | ViT-T initialized from ViT-S · CIFAR-100 | test accuracy: 81.4 -> 79.5 | Hurt |
| Use magnitude pruning instead of weight selection | ViT-T initialized from ViT-S · CIFAR-100 | test accuracy: 81.4 -> 73.8 | Hurt |
| Use a larger teacher model | Weight selection for ViT-T · CIFAR-100 | test accuracy: 81.4 -> 77.6 -> 76.9 | Hurt |
| Remove attention block weights from initialization (ablation) | Weight selection on ViT-T · CIFAR-100 | test accuracy: 81.4 -> 77.3 | Hurt |
Surprising findings
- A 5M-parameter subset of ViT-L (301M parameters) still improves ViT-T accuracy by 4.5 points, and smaller teachers give better initialization.
- Consistent index selection across all weight tensors matters greatly: random selection without consistency drops CIFAR-100 accuracy from 81.7 to 77.4 for ViT-T.
Limitations
- Only evaluated on image classification with ViT and ConvNeXt families; generality to language or other modalities is untested in the paper.
- Layer selection preference reverses for a large teacher (ViT-L to ViT-S), where uniform selection is slightly better, so the default first-N choice depends on the teacher-student size ratio.
- Pruning baselines are adapted to this setting and may not represent their best use.
Experimental setup
Datasets: ImageNet-1K ImageNet-21K CIFAR-10 CIFAR-100 Flowers Pets STL-10 Food-101 DTD SVHN EuroSAT
Benchmarks: ImageNet-1K CIFAR-10 CIFAR-100 STL-10 Food-101 Flowers Pets DTD SVHN EuroSAT
Metrics: test accuracy training epochs to reach accuracy linear probing accuracy
Baselines: timm default initialization Xavier initialization Kaiming initialization L1 pruning magnitude pruning mimetic initialization logit-based knowledge distillation feature-based knowledge distillation
Builds on: ConvNeXt training recipe timm knowledge distillation DeiT
Closest papers by meaning
- WAVE: Weight Templates for Adaptive Initialization of Variable-sized Models (CVPR 2025 · similarity 0.87)
- Where Do Large Learning Rates Lead Us? (NeurIPS 2024 · similarity 0.86)
- Pay Attention to Small Weights (NeurIPS 2025 · similarity 0.85)
- Diffusion-based Neural Network Weights Generation (ICLR 2025 · similarity 0.85)
- Sinusoidal Initialization, Time for a New Start (NeurIPS 2025 · similarity 0.85)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.