Research map / Architectures and efficiency / RNNs and normalization
Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends the mirror flow framework to include explicit regularization, showing it induces a time-dependent Legendre function that shapes implicit bias (positional, type, range shrinking), and proposes switching off weight decay during training to improve generalization.
Problem
How explicit regularization (e.g., weight decay) interacts with the implicit bias of overparameterized models, which is usually treated as constant and separately from explicit regularization.
Approach
Incorporates explicit regularization into the mirror flow framework via commuting reparameterizations, yielding a time-dependent Legendre/Bregman function whose shape and domain depend on the accumulated regularization; this lasting effect is exploited by turning off weight decay mid-training.
Key results
- In matrix sensing, turning off weight decay at T=675 recovers the sparse ground truth (reconstruction error below 1e-2), whereas constant weight decay does not.
- For a ViT fine-tuned on ImageNet, turning off weight decay at epoch 150 can yield more than 1% improved validation accuracy at similar relative sparsity.
- Constant weight decay trajectories show increasing weight decay lowers the nuclear-to-Frobenius norm ratio, indicating a shift from L2 toward L1-like bias.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Switch off weight decay after half the training time instead of keeping constant weight decay at 0.01 | Quadratic matrix sensing · Matrix sensing | Reconstruction error: $3.9 \times 10^{-2}$ -> $1.7 \times 10^{-4}$ | Helped |
| Use linear decay with matched total regularization, switched off after half the training time (ablation) | Quadratic matrix sensing · Matrix sensing | Reconstruction error: $3.9 \times 10^{-2}$ -> $2.3 \times 10^{-4}$ | Helped |
| Use cosine decay with matched total regularization, switched off after half the training time (ablation) | Quadratic matrix sensing · Matrix sensing | Reconstruction error: $3.9 \times 10^{-2}$ -> $2.1 \times 10^{-4}$ | Helped |
| Increase weight decay to 0.2 before switching it off (ablation) | Quadratic matrix sensing · Matrix sensing | Reconstruction error: $3.9 \times 10^{-2}$ -> $2.7 \times 10^{-5}$ | Helped |
| Increase weight decay to 2 before switching it off (ablation) | Quadratic matrix sensing · Matrix sensing | Reconstruction error: $3.9 \times 10^{-2}$ -> $2.4 \times 10^{-5}$ | Helped |
| Increase weight decay to 20 before switching it off (ablation) | Quadratic matrix sensing · Matrix sensing | Reconstruction error: $3.9 \times 10^{-2}$ -> $1.4 \times 10^{-6}$ | Helped |
| Turn weight decay off during training rather than following constant weight-decay trajectories | ViT fine-tuning · ImageNet validation | accuracy: more than 1% | Helped |
Surprising findings
- Switching weight decay off mid-training lets the implicit bias retain the effect of past regularization, reaching sparser solutions than constant weight decay.
Limitations
- Theory applies only to commuting reparameterizations (e.g., up to depth two); deeper reparameterizations with weight decay are excluded.
- Attention and LoRA analyses require additional assumptions such as alignment and fixed value matrix, so experiments extend beyond proven settings.
- Linear reparameterization with L1 cannot recover the ground truth after switch-off, showing the effect is specific to quadratic parameterizations.
Experimental setup
Datasets: tiny_shakespeare ImageNet synthetic matrix sensing data
Benchmarks: tiny_shakespeare ImageNet
Metrics: reconstruction error training loss nuclear-to-Frobenius norm ratio validation accuracy
Baselines: constant weight decay linear reparameterization with L1 regularization linear and cosine weight decay decay schedules
Builds on: mirror flow framework hyperbolic entropy implicit bias commuting parameterizations (Gunasekar et al.) LoRA
Closest papers by meaning
- Mask in the Mirror: Implicit Sparsification (ICLR 2025 · similarity 0.91)
- Estimating Implicit Regularization in Deep Learning (NeurIPS 2026 · similarity 0.89)
- Implicit Bias of Mirror Flow for Shallow Neural Networks in Univariate Regression (ICLR 2025 · similarity 0.89)
- Implicit Regularization Paths of Weighted Neural Representations (NeurIPS 2024 · similarity 0.88)
- Effects of Momentum in Implicit Bias of Gradient Flow for Diagonal Linear Networks (AAAI 2025 · similarity 0.87)
- Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.