Research map / Optimization / Implicit bias and generalization
Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends differentiable L1 weight factorization to deep (depth-D) factorizations, proving equivalence to L_{2/D} sparse regularization, with a tailored initialization and learning-rate guidance for SGD-based sparse training.
Problem
Sparse regularization (L1) is non-differentiable and incompatible with SGD in neural networks; shallow weight factorization addresses this but its extension to deeper factorizations and their training dynamics/initialization are not understood.
Approach
Factorize every weight into D>=2 scalar factors and apply L2 weight decay to the factors; this is equivalent to non-convex L_{2/D} regularization on the collapsed weights, with a variance-matched, truncated initialization and large-learning-rate schedules to make deep factorizations trainable.
Key results
- For LeNet-300-100 on MNIST, DWF with D=3,4 stays within 5% of dense accuracy above compression ratio 500, significantly surpassing other pruning methods.
- For VGG-19 on CIFAR100 at 10% tolerance, DWF with D=3 reaches compression ratio 1014, improving 8% over D=2 and 381% over SynFlow.
- The best DWF model achieves 2 to 5 times the compression ratio of the best pruning method across 10 tested scenarios, and DWF with D=3,4 achieves highest compression in 9 of 10 cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DWF instead of vanilla L1 regularization with SGD | LeNet-300-100 · Fashion-MNIST, at 80% test accuracy | compression ratio | Helped |
| Increase factorization depth from 2 to 3 (ablation) | DWF · Fashion-MNIST, at the same test accuracy | compression ratio: up to four times sparser | Helped |
| Use DWF rather than vanilla L1, followed by magnitude pruning | LeNet-300-100 · Fashion-MNIST, at 80% test accuracy | parameter count: twice as many parameters; three times as many | Helped |
| Use depth 3 or 4 instead of depth 2 (ablation) | DWF · LeNet-300-100, MNIST, compression ratio above 500 | accuracy relative to dense model: within 5% | Helped |
| Use the tailored DWF initialization rather than standard factor initialization (ablation) | DWF with depth greater than 2 · Initialization and training experiments | accuracy | Helped |
| Add interval truncation to variance-matched factor initialization (ablation) | DWF initialization · Deep factorization | initialization quality | Helped |
| Use variance matching without interval truncation (ablation) | DWF initialization · Depth greater than 2 | performance | Hurt |
| Use standard factor initialization instead of variance-matched initialization (ablation) | DWF · Factorized neural networks | activation variance | Hurt |
| Use a sufficiently large learning rate instead of a too-small learning rate (ablation) | DWF · LeNet-300-100, depth 4 | sparsity | Helped |
| Increase regularization strength (ablation) | DWF · Across factorization depths | compression ratio | Helped |
| Increase factorization depth at fixed regularization strength (ablation) | DWF · Across architectures and datasets | compression ratio | Helped |
| Increase factorization depth at fixed regularization strength (ablation) | DWF · Across architectures and datasets | generalization performance | Hurt |
Surprising findings
- Standard (Kaiming/LeCun-type) initialization fails for factorized networks with D>2, causing vanishing activation variance and dead weights, which motivates the truncated DWF initialization.
- Increasing factorization depth beyond four did not yield further sparsity improvements and introduced optimization challenges.
Limitations
- Only unstructured sparsity is studied; structured sparsity is left for future work.
- Experiments focus on D in {2,3,4}; larger depths gave no further gains and caused optimization difficulties.
- Experiments are limited to small/medium CV architectures and datasets.
Experimental setup
Datasets: MNIST Fashion-MNIST Kuzushiji-MNIST CIFAR10 CIFAR100 Tiny ImageNet
Benchmarks: MNIST Fashion-MNIST Kuzushiji-MNIST CIFAR10 CIFAR100 Tiny ImageNet
Metrics: test accuracy compression ratio sparsity
Baselines: Global magnitude pruning SNIP SynFlow Random pruning Shallow factorization (D=2) Vanilla L1 regularization
Builds on: shallow weight factorization (differentiable L1) spred SNIP SynFlow
Closest papers by meaning
- Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning (ICLR 2021 · similarity 0.90)
- Implicit Regularization in Deep Matrix Factorization (NeurIPS 2019 · similarity 0.90)
- Diving into the shallows: a computational perspective on large-scale shallow learning (NeurIPS 2017 · similarity 0.90)
- Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorization (ICLR 2025 · similarity 0.89)
- Implicit Regularization for Tubal Tensor Factorizations via Gradient Descent (ICML 2025 · similarity 0.88)
- Deep linear networks for regression are implicitly regularized towards flat minima (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.