Research map / Architectures and efficiency / Pruning and efficient tuning
Transferring Learning Trajectories of Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes learning transfer: synthesizing a target learning trajectory from a source trajectory by finding a permutation symmetry via successive gradient matching (GMT/FGMT), with linear trajectories and gradient caching for efficiency.
Problem
Training DNNs is computationally expensive, and duplicated or similar training runs (ensembles, fine-tuning) repeat work; existing learning trajectories are not reused for training from a different initialization.
Approach
Treat a learning trajectory from one initialization as transferable to another initialization by applying a permutation symmetry to its parameter updates, found by matching gradients successively along the trajectory via a linear assignment (coordinate descent) approach, with linear-trajectory and gradient-caching approximations for storage and compute savings.
Key results
- In the random initialization scenario, GMT/FGMT closely approximate the Oracle baseline's transferred accuracy on MNIST, CIFAR-10 and ImageNet, achieving non-trivial accuracy without any direct training.
- In the pre-trained scenario, Naive transfer always fails, while the peak accuracies of GMT/FGMT largely outperform Naive.
- Transferred parameters (especially in pre-trained scenarios) train faster than standard training, and the transferred parameter inherits the target initialization's basin and better generalization even when the source is poorly generalizing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| transfer the trajectory using GMT | randomly initialized models · MNIST, CIFAR-10, and ImageNet | validation accuracy | Helped |
| use FGMT with cached gradients instead of GMT | trajectory transfer · random and pre-trained initialization scenarios | validation accuracy | No clear effect |
| replace identity-permutation transfer with GMT or FGMT | pre-trained initialization transfer · CIFAR-10 to CIFAR-100, ImageNet to Cars, and ImageNet to CUB | peak validation accuracy | Helped |
| initialize subsequent training from a transferred parameter | standard training from random or pre-trained initialization · CIFAR-10, CIFAR-10 to CIFAR-100, ImageNet to Cars, and ImageNet to CUB | validation accuracy | Helped |
| initialize subsequent training from GMT/FGMT or Oracle transfer rather than Naive transfer | pre-trained initialization transfer · CIFAR-10 to CIFAR-100, ImageNet to Cars, and ImageNet to CUB | training convergence | Helped |
| use a linearly interpolated source trajectory instead of actual training checkpoints (ablation) | trajectory transfer · CIFAR-10 (Conv8) | stability of transferred accuracy | Helped |
| transfer from a poorly generalizing source initialization to a better-generalizing target initialization | pre-trained model trajectory transfer · Cars and CUB | generalization ability | Helped |
| transfer to the target initialization and fine-tune, rather than simply permuting the source parameter | pre-trained initialization · CIFAR-10 to CIFAR-100, ImageNet to Cars, and ImageNet to CUB | loss-landscape basin similarity to Target | Helped |
Surprising findings
- Transferring a trajectory from a poorly generalizing pre-trained initialization to a better one yields parameters that inherit the better generalization of the target initialization.
- Linear trajectories transfer more stably with less variance than actual training trajectories.
Limitations
- Assumption (P) holds weakly or may break after hundreds of iterations for modern architectures.
- Pre-trained scenario: accuracy of the transferred parameters deteriorates after the middle of the timestep, requiring early stopping.
- Theoretical validation of the assumption is limited to 2-layer ReLU networks at initialization.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100 ImageNet Stanford Cars CUB-200-2011
Benchmarks: MNIST CIFAR-10 CIFAR-100 ImageNet Stanford Cars CUB-200-2011
Metrics: validation accuracy cosine similarity between trajectories generalization gap loss barrier along linear path
Baselines: Naive (identity permutation) Oracle (permutation from actually trained target) Standard Training
Builds on: Git Re-basin linear mode connectivity monotonic linear interpolation task arithmetic
Closest papers by meaning
- Features are fate: a theory of transfer learning in high-dimensional regression (ICML 2025 · similarity 0.86)
- A Gift From Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning (CVPR 2017 · similarity 0.85)
- Towards Stable and Storage-efficient Dataset Distillation: Matching Convexified Trajectory (CVPR 2025 · similarity 0.85)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.85)
- The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions (ICML 2025 · similarity 0.84)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.