Research map / Architectures and efficiency / Pruning and efficient tuning
Cautious Weight Decay
Lizhang Chen, Jonathan Li, Kaizhao Liang, Baiyu Su, Cong Xie, Chen Liang, Ni Lao, qiang liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply weight decay only where update and parameter signs align (CWD) | AdamW · OLMo-1B 100B tokens | validation loss: 2.65 -> 2.56 | Improved |
| apply CWD | Muon · OLMo-1B 100B tokens | validation loss: 2.51 -> 2.42 | Improved |
| replace sign mask with random Bernoulli mask of matched sparsity (ablation) | AdamW+CWD · OLMo-1B 100B tokens | validation loss: 2.56 -> 2.82 | Hurt |
| replace update-based mask with gradient-based mask (ablation) | Muon+CWD · OLMo-1B 100B tokens | validation loss: 2.42 -> 2.74 | Hurt |
| remove weight decay entirely (lambda=0) (ablation) | AdamW · OLMo-1B 100B tokens | validation loss: 2.65 -> 2.70 | Hurt |
| apply CWD to AdamW, Lion, Muon without retuning | ViT-S/16 · ImageNet validation | accuracy (%): 78.84 -> 79.45 (AdamW) | Improved |
| apply CWD to Muon | ViT-B/16 · ImageNet validation | accuracy (%): 80.83 -> 81.04 | Improved |
| apply CWD to AdamW at 1B scale | OLMo-1B · downstream zero-shot | HellaSwag acc_norm: 0.38 -> 0.40 | Improved |
| apply CWD to Muon at 1B scale | OLMo-1B · downstream zero-shot | ARC-Easy acc_norm: 0.51 -> 0.51 | No clear effect |
| use elementwise CWD instead of LoRA baseline | TinyLlama · MMLU 5-shot | accuracy (%): 25.81 -> 26.42 | Improved |
| use elementwise CWD instead of SPD | Mistral-7B · WinoGrande 5-shot | accuracy (%): 78.81 -> 78.92 | No clear effect |
| use layerwise inner-product mask instead of elementwise mask (ablation) | CWD on Mistral-7B · MMLU 5-shot | accuracy (%): 62.13 -> 61.76 | Hurt |
Closest papers by meaning
- Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models (NeurIPS 2024 · similarity 0.90)
- Dynamics-Aware Preference Optimization for Vision-Language Models (CVPR 2026 · similarity 0.87)
- Why Do We Need Weight Decay in Modern Deep Learning? (NeurIPS 2024 · similarity 0.87)
- Investigating the Role of Weight Decay in Enhancing Nonconvex SGD (CVPR 2025 · similarity 0.87)
- MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization (NeurIPS 2025 · similarity 0.86)
- Implicit Bias of AdamW: $\ell_\infty$-Norm Constrained Optimization (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
