Research map / Optimization / Implicit bias and generalization
Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity
Amir Joudaki, Giulia Lanzillotta, Mohammad Samragh, Seyed Iman Mirzadeh, Keivan Alizadeh, Thomas Hofmann, Mehrdad Farajtabar, Fartash Faghri
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| inject Gaussian gradient noise (relative magnitude 0.01) after cloning | SGD on cloned MLP · Cloning MLP experiment | escape from cloning manifold: single step noise 0.01 relative to gradient norm | Improved |
| add dropout to cloned network (ablation) | cloned MLP · Cloning MLP experiment | escape from cloning manifold | Improved |
| add noise perturbation to cloned ViT | cloned ViT · Cloning ViT experiment | distance from manifold | No clear effect |
| add normalization layers (BN/LN) (ablation) | MLP/CNN/ResNet/ViT · Continual Tiny ImageNet (40 5-class tasks) | effective rank | Improved |
| add normalization to already cloned network (ablation) | cloned network · Cloning experiments | breaking clones | No clear effect |
| switch from SGD to Continual Backpropagation in second half | SGD · Bit-flipping (5M samples) | representation rank and online loss | Improved |
| add dropout in continual bit-flipping (ablation) | network · Bit-flipping | plasticity | Mixed |
Closest papers by meaning
- Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking (NeurIPS 2025 · similarity 0.88)
- A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks (ICML 2025 · similarity 0.87)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.86)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.86)
- Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
