Research map / Architectures and efficiency / RNNs and normalization
Understanding and Leveraging the Learning Phases of Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically and theoretically analyzes the learning phases of neural networks via layer reconstruction loss (instead of mutual information), finds three phases, proves them for a linear single-layer model, and proposes stopping classifier pre-training early for better transfer learning.
Problem
Learning dynamics of deep networks remain poorly understood; information bottleneck claims of fitting and compression phases are debated and hard to measure rigorously.
Approach
Measure a layer's reconstruction ability of the input (via a trained linear/auto-encoder decoder) alongside accuracy over training, use an empirically grounded data model with strong/weak features to prove the existence of reconstruction-loss phases for single-layer linear networks, and use this to motivate early stopping of pre-training for transfer learning.
Key results
- Across VGG-11, ResNet-10 and a single dense-layer network on CIFAR-10/100, Fashion-MNIST and MNIST, reconstruction loss shows three phases: near-constant, decrease, then increase, with phases most visible in layers near the output.
- For a linear single-layer model with a strong/weak feature data model, the authors prove the reconstruction error decreases from Omega(k^2/l) to O(k^2/l^2) and then increases again to Omega(k^2/l), with approximation error of the reconstruction function bounded by 2.
- In transfer experiments, downstream accuracy peaks at roughly the same training iterations where the reconstruction loss is minimal, before the pre-training task performance is optimal, supporting early stopping of pre-training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Continue classifier training after early training | Linear classifier on the theoretical data model · Theoretical data model | reconstruction error: R^{(0)}=\Omega(k^2/l) to R^{(t)}=O(k^2/l^2); increases to R^{(t\rightarrow\infty)}=\Omega(k^2/l) | Mixed |
| Stop pre-training before source-task performance is optimal | VGG or ResNet classifier used for transfer learning · Fashion-MNIST ↔ MNIST; CIFAR-10 ↔ CIFAR-100 | downstream accuracy | Helped |
| Use later training checkpoints rather than an earlier checkpoint | VGG-11, ResNet-10, and F0 · CIFAR-10/100, Fashion-MNIST, and MNIST | reconstruction loss | Mixed |
Surprising findings
- Reconstruction ability of intermediate representations can worsen even as classification accuracy keeps improving during training.
- Pre-training should stop well before the pre-training task performance is optimal to maximize downstream transfer performance.
Limitations
- Theoretical analysis is restricted to a single linear layer with deterministic initialization; extension to deep networks is unclear.
- The data model is an assumption that is only empirically verified; the analysis relies on asymptotic O-notation with large n, d, l.
- Transfer learning evidence is limited to a few dataset pairs and a linear probe setup; when to stop depends on dataset similarity.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Fashion-MNIST MNIST
Benchmarks: CIFAR-10 CIFAR-100 Fashion-MNIST MNIST
Metrics: reconstruction loss accuracy normalized accuracy and reconstruction loss downstream transfer accuracy
Baselines: information bottleneck analysis
Builds on: information bottleneck Johnson-Lindenstrauss lemma neural tangent kernel
Closest papers by meaning
- Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression (ICLR 2024 · similarity 0.89)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.87)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.87)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.87)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.87)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.