Research map / Optimization / Implicit bias and generalization
Mask in the Mirror: Implicit Sparsification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Explains why continuous sparsification with m⊙w parameterization outperforms explicit L1 via a time-dependent mirror flow (implicit L2→L1 bias controlled by dynamic weight decay), and proposes PILoT, which improves over STR and spred on CIFAR and ImageNet.
Problem
Continuous sparsification methods using mask-weight parameterization m⊙w work better than explicit L1 regularization, but the reason is not understood; it is also unclear how to control the implicit sparsity bias during training to reach high sparsity while preserving accuracy.
Approach
Shows that m⊙w with time-varying weight decay induces a time-dependent mirror flow with a time-dependent Bregman potential (hyperbolic-entropy-like), so the implicit bias moves from L2 to L1 over training; uses this to design PILoT with a non-balanced initialization enabling sign flips and an adaptive decaying regularization schedule.
Key results
- On ImageNet ResNet-50 at about 98.2% sparsity, PILoT reaches 64.06% top-1 versus 61.46% for STR and 62.71% for spred.
- On ImageNet ResNet-50 at about 96% sparsity, PILoT reaches 70.49% top-1 versus 70.4% for STR at 95.03%, and 69.47% for spred at 94.5%.
- In the diagonal linear network experiment, PILoT with decaying regularization reaches the ground truth, while spred cannot due to its inability to sign flip.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dynamic regularization instead of constant regularization | PILoT · diagonal linear network | distance to ground truth | Helped |
| Use the PILoT initialization instead of the spred initialization (ablation) | continuous sparsification · diagonal linear network | distance to ground truth | Helped |
| Use decaying regularization schedules instead of constant regularization (ablation) | PILoT · diagonal linear network | distance to ground truth | Helped |
| Use decaying rather than constant regularization schedules (ablation) | spred and LASSO · diagonal linear network | distance to ground truth | Hurt |
| Add the PILoT initialization without dynamic regularization (ablation) | spred-style continuous sparsification · CIFAR10 | accuracy | Helped |
| Use PILoT instead of spred | ResNet-50 · ImageNet at 80.00% sparsity | Top-1 accuracy: 75.5 -> 75.62 | Helped |
| Use PILoT instead of spred | ResNet-50 · ImageNet at 97.19% sparsity | Top-1 accuracy: 66.12 -> 68.49 | Helped |
| Use PILoT instead of STR | ResNet-50 · ImageNet at high sparsity | Top-1 accuracy: 61.46 -> 66.49 | Helped |
| Use PILoT instead of STR | continuous sparsification · CIFAR10 and CIFAR100, high sparsity | accuracy | Helped |
| Use spred instead of STR | continuous sparsification · CIFAR10 and CIFAR100, high sparsity | accuracy | Helped |
| Use the m ⊙ w parameterization with PILoT in iterative pruning | LRR and WR · ImageNet ResNet-18 | accuracy | Helped |
Surprising findings
- Spred's gradient flow cannot sign flip, so it cannot reach the optimal solution, which is why constant regularization works best for spred but decaying regularization works best for PILoT.
Limitations
- The implicit L1 limit requires exponentially small scaling, which can cause numerical problems and saddle-point slowdowns.
- The convergence constant B can be large, so the guarantees may be loose.
- Theoretical optimality is shown only for diagonal linear networks and underdetermined linear regression.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet synthetic Gaussian regression data
Benchmarks: CIFAR-10 CIFAR-100 ImageNet
Metrics: top-1 accuracy sparsity distance to ground truth (diagonal linear network)
Baselines: STR spred LASSO Weight Rewinding (WR) Learning Rate Rewinding (LRR)
Builds on: spred mirror flow framework (Gunasekar et al. style implicit bias) STR LASSO
Closest papers by meaning
- Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias? (ICML 2025 · similarity 0.91)
- Implicit Sparse Regularization: The Impact of Depth and Early Stopping (NeurIPS 2021 · similarity 0.88)
- Implicit Bias of Mirror Flow for Shallow Neural Networks in Univariate Regression (ICLR 2025 · similarity 0.88)
- Implicit Regularization for Tubal Tensor Factorizations via Gradient Descent (ICML 2025 · similarity 0.87)
- Estimating Implicit Regularization in Deep Learning (NeurIPS 2026 · similarity 0.86)
- Improving Generalization and Convergence by Enhancing Implicit Regularization (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.