Research map / Learning theory and algorithms / Matrix and tensor methods
Learning Neural Networks with Adaptive Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an adaptive matrix-variate normal prior on the final-layer weights | MTL · SARCOS tasks 1–7 | explained variance: 0.4418 -> 0.4769; 0.3472 -> 0.3969; 0.5222 -> 0.5485; 0.5036 -> 0.5308; 0.6024 -> 0.6202; 0.4727 -> 0.5085; 0.5298 -> 0. | Helped |
| Use AdaReg instead of weight decay | SARCOS MTL · SARCOS | stable rank: 4.83 -> 2.88 | Helped |
| Use AdaReg instead of weight decay | SARCOS MTL · SARCOS | spectral norm: 0.92 -> 0.70 | Helped |
| Apply AdaReg to the softmax layer rather than applying it across more network layers (ablation) | MNIST CNN · MNIST, batch size 2048 | generalization | Helped |
| Add AdaReg to batch normalization | MNIST · MNIST | generalization | Helped |
| Add AdaReg to dropout | MNIST · MNIST | generalization | No clear effect |
| Apply AdaReg to the final softmax layer | MNIST CNN · MNIST training size 600 | accuracy: 89.34 -> 92.50 | Helped |
| Apply AdaReg to the final softmax layer | MNIST CNN · MNIST training size 60000 | accuracy: 98.99 -> 99.19 | Helped |
Closest papers by meaning
- Function-Space Regularization in Neural Networks: A Probabilistic Perspective (ICML 2023 · similarity 0.89)
- Adaptive Knowledge Driven Regularization for Deep Neural Networks (AAAI 2021 · similarity 0.88)
- Learning Sparse Neural Networks through L_0 Regularization (ICLR 2018 · similarity 0.87)
- Cauchy-Schwarz Regularizers (ICLR 2025 · similarity 0.86)
- Generalization in Kernel Regression Under Realistic Assumptions (ICML 2024 · similarity 0.85)
- Regularized Adaptive Momentum Dual Averaging with an Efficient Inexact Subproblem Solver for Training Structured Neural Network (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.