Research map / Architectures and efficiency / RNNs and normalization
Cross-regularization: Adaptive Model Complexity through Validation Gradients
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes cross-regularization, which adapts regularization parameters (norms, noise scales, augmentation magnitudes) by gradient descent on a held-out regularization set during single-run training, with claimed convergence guarantees.
Problem
Choosing regularization strength requires manual tuning or repeated cross-validation runs; existing adaptive methods either need inverse Hessians, parameter history, multiple runs, or optimize proxy training objectives.
Approach
Split optimization: model parameters are trained on training data while regularization parameters receive gradients of validation loss on a separate regularization set, with validation inference averaging Monte Carlo noise samples so that noise scales are identifiable; claims convergence to cross-validation optima.
Key results
- On VGG-16 CIFAR-10, cross-regularization reaches 83.7% accuracy vs 76.0% baseline in a single run, matching PBT (83.7%) with claimed ~10x less compute.
- Learned layer noise reaches sigma up to about 12-14 standard deviations, which the paper relates to ~99% dropout-equivalent rates.
- Calibration: final ECE 0.038 for X-Reg vs 0.030 for a 5-member Deep Ensemble and 0.163 for the uncalibrated model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use validation-gradient noise adaptation instead of the baseline regularization | VGG-16 · CIFAR-10 | accuracy: 76.0% -> 83.7% | Helped |
| Replace cross-regularization with PBT noise optimization | VGG-16 · CIFAR-10 | accuracy: 83.7% vs 83.7% | No clear effect |
| Add validation-guided adaptive augmentation | Cross-regularization · SVHN | test accuracy: 82.8% to 86.3% | Helped |
| Add validation-guided adaptive augmentation | Cross-regularization · SVHN | generalization gap: 16.2% to 7.3% | Helped |
| Use fixed noise (σ=1 in the final five layers) instead of adaptive noise (ablation) | VGG-16 · CIFAR-10 | generalization gap | Hurt |
| Use variational dropout instead of cross-regularization for layer-wise noise adaptation (ablation) | Neural network noise regularization · Neural network experiments | noise adaptation stability | Hurt |
| Reduce validation Monte Carlo samples below 3-5 (ablation) | Cross-regularization · Neural network experiments | performance | Hurt |
| Update regularization parameters sparsely (every 30 steps) (ablation) | Cross-regularization · Neural network experiments | convergence | No clear effect |
| Reduce the regularization set to 1% of training data (ablation) | Cross-regularization · Neural network experiments | performance | No clear effect |
| Replace additive noise with multiplicative noise (ablation) | Cross-regularization · Neural network experiments | noise-scale interpretability | Hurt |
| Use a global noise scale instead of layer-wise noise (ablation) | Neural network noise regularization · Neural network experiments | architecture-specific regularization | Hurt |
| Use per-unit noise instead of layer-wise noise (ablation) | Neural network noise regularization · Neural network experiments | regularization parameter efficiency | Hurt |
Surprising findings
- Learned noise levels far exceed conventional regularization strengths (sigma around 12-14), yet the networks remain functional.
- Standard variational dropout collapsed to zero noise or became unstable in the authors' attempts.
Limitations
- Convergence theorems rely on strong assumptions (strong convexity in rho, Lipschitz coupling) that are asserted for neural networks rather than verified.
- Neural network experiments are small-scale (VGG-16/WRN on CIFAR-10, SVHN) with single-run results and no reported variance beyond shaded CIs in one figure.
- Appendix proofs contain gaps (e.g., cross-validation equivalence proof is essentially definitional; step-size conditions are heuristic).
Experimental setup
Datasets: CIFAR-10 SVHN diabetes progression dataset synthetic regression data
Benchmarks: CIFAR-10 SVHN diabetes progression dataset
Metrics: test accuracy validation loss generalization gap expected calibration error
Baselines: Population Based Training Fixed noise regularization Temperature Scaling Deep Ensemble Ridge regression with cross-validation LASSO Variational dropout
Builds on: Population Based Training Concrete Dropout variational dropout AutoAugment
Closest papers by meaning
- Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters (ICML 2016 · similarity 0.87)
- Implicit Gradient Regularization (ICLR 2021 · similarity 0.86)
- Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization (NeurIPS 2017 · similarity 0.86)
- Characterizing the Edge of Stability in Variational Training Without Priors (NeurIPS 2026 · similarity 0.86)
- The Effects of Regularization and Data Augmentation are Class Dependent (NeurIPS 2022 · similarity 0.85)
- Improving Adaptivity via Over-Parameterization in Sequence Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.