Research map / Architectures and efficiency / RNNs and normalization
Initialization and Regularization of Factorized Neural Layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use both spectral initialization and Frobenius decay instead of default low-rank training (ablation) | ResNet32x2 low-rank layers · CIFAR-10, compression 0.10 | accuracy: 92.97 -> 94.34 | Helped |
| Use both spectral initialization and Frobenius decay instead of default low-rank training (ablation) | ResNet32x2 low-rank layers · CIFAR-100, compression 0.10 | accuracy: 72.71 -> 74.41 | Helped |
| Use both spectral initialization and Frobenius decay instead of default low-rank training (ablation) | ResNet32x2 low-rank layers · Tiny-ImageNet, compression 0.15 | accuracy: 60.82 -> 62.24 | Helped |
| Apply spectral initialization and Frobenius decay together rather than either alone (ablation) | low-rank ResNet models · CIFAR-10 | accuracy: 91.58; 92.90; 94.34 | Mixed |
| Use spectral initialization and Frobenius decay for low-rank convolution layers | VGG19-NFC · CIFAR-10, compression 0.10 | accuracy: 94.14 -> 91.58 | Hurt |
| Apply Frobenius decay and spectral initialization to low-rank WideResNet (ablation) | WideResNet28-10 · CIFAR-10, compression 0.06667 | test error: 5.31±0.06 -> 3.86±0.22 | Helped |
| Use Frobenius decay with an overcomplete full factorization instead of ordinary weight decay (ablation) | ResNet56 · CIFAR-10 | test accuracy: 92.95 -> 94.13 | Helped |
| Use Frobenius decay with an overcomplete wide factorization instead of ordinary weight decay (ablation) | ResNet56 · CIFAR-100 | test accuracy: 70.69 -> 73.01 | Helped |
| Use an overcomplete factorization with Frobenius decay rather than the unfactorized model | ResNet110 · CIFAR-10 | test accuracy: 93.61 -> 94.42 | Helped |
| Use an overcomplete factorization with Frobenius decay rather than the unfactorized model | ResNet110 · CIFAR-100 | test accuracy: 72.11 -> 74.17 | Helped |
| Use tensor analogs of spectral initialization and Frobenius decay (ablation) | Tensor-Train WideResNet28-10 · CIFAR-10, compression 0.01667 | test error: 8.55±4.22 -> 6.22±0.50 | Helped |
Closest papers by meaning
- Implicit Regularization for Tubal Tensor Factorizations via Gradient Descent (ICML 2025 · similarity 0.88)
- Geometry-aware training of factorized layers in tensor Tucker format (NeurIPS 2024 · similarity 0.88)
- Weight decay induces low-rank attention layers (NeurIPS 2024 · similarity 0.87)
- Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries (ICLR 2025 · similarity 0.87)
- IDInit: A Universal and Stable Initialization Method for Neural Network Training (ICLR 2025 · similarity 0.87)
- Sparser, Better, Deeper, Stronger: Improving Static Sparse Training with Exact Orthogonal Initialization (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.