Research map / Architectures and efficiency / Pruning and efficient tuning
Improving Deep Learning Optimization through Constrained Parameter Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Constrained Parameter Regularization (CPR), which replaces uniform weight decay with per-parameter-matrix upper bounds on L2 norm enforced via an adapted augmented Lagrangian method, with four bound-initialization strategies.
Problem
Weight decay applies one uniform penalty coefficient to all parameters and is static, which may be too restrictive for some parameter matrices and insufficient for others; choosing its value is also task-sensitive.
Approach
Formulate regularization as a constrained problem with an upper bound on a statistical measure (e.g., squared L2 norm) of each parameter matrix, solved by an augmented-Lagrangian-style multiplier update performed every step, so each matrix gets an individual, dynamically adapting regularization strength. Bounds are initialized from warm-start norms or the first inflection point of the regularizer,
Key results
- On OpenWebText GPT2s, AdamCPR (Kappa-IP) reaches perplexity 17.94 vs 18.23 for best AdamW (wd 1e-2), and CPR saves up to 33% training budget at that scale.
- On ImageNet DeiT-Small, AdamCPR with Kappa-IP reaches 79.84% top-1 vs 79.16% for best AdamW; DeiT-Base Kappa-WS reaches 81.19% vs 80.56%.
- Medical segmentation with SGD+CPR improves Dice from 83.99 to 84.23 (BTCV), 92.92 to 93.18 (Heart), and 76.22 to 76.65 (BraTS).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace AdamW weight decay with AdamCPR using Kappa-WS or Kappa-IP | DeiT pretraining · ImageNet | Top-1 accuracy: 79.16 -> 79.81; 79.16 -> 79.84; 80.56 -> 81.19; 80.56 -> 80.95 | Helped |
| Use Kappa-WS bound initialization at 2x learning-rate warmup instead of the best AdamW weight decay | CLIP ViT-B/32 fine-tuning · ImageNet | Top-1 accuracy: 75.39 -> 75.52 | Helped |
| Use Kappa-IP bound initialization instead of AdamW | CLIP ViT-B/32 fine-tuning · ImageNet | Top-1 accuracy: 75.39 -> 75.40 | No clear effect |
| Replace AdamW with AdamCPR using Kappa-IP | GPT2s · OpenWebText | Perplexity: 18.23 $\pm$ 0.0113 -> 17.94 | Helped |
| Replace weight decay with SGDCPR using Kappa-WS | nnU-Net · BTCV, Heart Segmentation, and BraTS 2020 | Dice score: 83.99 -> 84.23; 92.92 -> 93.18; 76.22 ->76.65 | Helped |
| Vary the multiplier update rate μ (ablation) | AdamCPR · CIFAR100 and OpenWebText | performance: 0.422; 0.423; 0.423; 0.423 accuracy; 20.91; 20.90; 20.90; 20.90 PPL | No clear effect |
| Use adaptive bounds (AdaCPR) instead of fixed bounds (ablation) | CPR · CIFAR100 | accuracy | No clear effect |
| Initialize bounds from initial parameter measures (Kappa-kI₀) instead of a shared uniform Kappa-K bound (ablation) | AdamCPR · CIFAR100 | accuracy | Helped |
| Initialize bounds with a warm start (Kappa-WS) instead of Kappa-kI₀ or Kappa-K (ablation) | AdamCPR · CIFAR100 | accuracy | Helped |
| Use Kappa-IP instead of Kappa-WS (ablation) | SGDCPR · CIFAR100 | accuracy | Hurt |
| Use Kappa-IP instead of Kappa-WS (ablation) | AdamCPR · PubMedQA fine-tuning with Mistral 7B and LoRA | answer accuracy change | Hurt |
| Use standard deviation instead of squared L2 norm as the regularization measure (ablation) | AdamCPR · CIFAR100 | accuracy | Hurt |
Surprising findings
- Adaptive bounds (AdaCPR) did not improve over fixed bounds in the reported experiments.
- Kappa-IP does not work with SGD, attributed to different convergence behavior compared to Adam.
Limitations
- Runtime overhead grows for larger models (up to ~6% for 1.1B parameters).
- Benefit diminishes when weight regularization has minimal impact, e.g., small models on large datasets.
- Kappa-IP underperforms for LoRA fine-tuning and SGD; no clear gain on noisy CIFAR100-C.
Experimental setup
Datasets: CIFAR100 CIFAR100-C ImageNet OpenWebText PubMedQA BTCV Medical Segmentation Decathlon Heart BraTS 2020
Benchmarks: CIFAR100 CIFAR100-C ImageNet OpenWebText perplexity PubMedQA TruthfulQA BTCV Medical Segmentation Decathlon Heart BraTS 2020
Metrics: top-1 accuracy perplexity Dice score percentage of correct labels change in accuracy
Baselines: AdamW AdaDecay Adaptive Weight Decay (AWD) AMOS Adam with rescaling SGD with weight decay
Builds on: AdamW augmented Lagrangian method nnU-Net WiSE-FT
Closest papers by meaning
- Learning Compact Neural Networks with Regularization (ICML 2018 · similarity 0.88)
- Explicitly Imposing Constraints in Deep Networks via Conditional Gradients Gives Improved Generalization and Faster Convergence (AAAI 2019 · similarity 0.88)
- Why Do We Need Weight Decay in Modern Deep Learning? (NeurIPS 2024 · similarity 0.87)
- AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning (NeurIPS 2026 · similarity 0.87)
- Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models (NeurIPS 2024 · similarity 0.86)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.