Research map / Architectures and efficiency / Pruning and efficient tuning
Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Selective Projection Decay (SPD), an Adam-compatible selective L2-SP/weight-decay variant that penalizes deviation from pre-trained weights only on layers whose recent updates are inconsistent, improving ID accuracy and OOD robustness for foundation-model fine-tuning.
Problem
Standard optimizers like AdamW, and uniform regularizers like L2-SP, either drift too far from the pre-trained initialization (hurting OOD robustness) or over-regularize all layers (hurting fitting) when fine-tuning foundation models.
Approach
Apply a projection-style penalty toward the pre-trained initialization only to layers meeting a selection condition c_t = -g_t^T(theta_{t-1}-theta_0) < 0 (inconsistent progress), with a deviation-ratio strength r_t derived from L2-SP written as a norm-ball projection; for PEFT it reduces to selective weight decay on LoRA factors toward zero.
Key results
- On ImageNet with CLIP ViT-Base, Adam-SPD reaches 84.21 ID and 50.13 OOD average, beating L2-SP (46.06 OOD avg) and FTP (49.65).
- On PASCAL-Context segmentation with Swin-Tiny, Adam-SPD improves OOD average by 36.5% over vanilla fine-tuning and 5.8% over L2-SP.
- On commonsense reasoning with LLaMA-7B LoRA, Adam-SPD averages 77.8 vs 74.7 for AdamW; with LLaMA-13B LoRA it reaches 82.2 vs 80.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace AdamW with Adam-SPD | CLIP ViT-Base fine-tuned on DomainNet · DomainNet Real fine-tune | OOD Avg. accuracy: 39.28 -> 45.93 | Helped |
| Replace AdamW with Adam-SPD | CLIP ViT-Base fine-tuned on DomainNet · DomainNet Clipart fine-tune | OOD Avg. accuracy: 36.36 -> 41.43 | Helped |
| Replace uniform L2-SP with selective SPD | CLIP ViT-Base fine-tuned on DomainNet Clipart · DomainNet Clipart fine-tune | OOD accuracy: 36.50 -> 41.43 | Helped |
| Replace Adam-SPD with uniform L2-SP regularization | CLIP ViT-Base fine-tuned on DomainNet Clipart · DomainNet Clipart fine-tune | OOD accuracy: 41.43 -> 36.50 | Hurt |
| Increase uniform L2-SP strength (ablation) | CLIP ViT-Base fine-tuned on DomainNet Clipart · DomainNet Clipart fine-tune | ID accuracy: 79.91 -> 27.25 | Hurt |
| Increase uniform L2-SP strength (ablation) | CLIP ViT-Base fine-tuned on DomainNet Clipart · DomainNet Clipart fine-tune | OOD accuracy: 41.34 -> 14.90 | Mixed |
| Increase SPD strength λ (ablation) | CLIP ViT-Base fine-tuned on DomainNet Clipart · DomainNet Clipart fine-tune | OOD accuracy: 41.34 -> 45.67 | Helped |
| Increase SPD strength λ (ablation) | CLIP ViT-Base fine-tuned on DomainNet Clipart · DomainNet Clipart fine-tune | ID accuracy: 79.91 -> 81.21 | No clear effect |
| Replace the ImageNet fine-tuning baseline with Adam-SPD | CLIP ViT-Base · ImageNet | ID accuracy: 83.66 -> 84.21 | Helped |
| Replace FTP with Adam-SPD | CLIP ViT-Base · ImageNet | OOD Avg. accuracy: 49.65 -> 50.13 | Helped |
| Replace vanilla fine-tuning with Adam-SPD | Swin ViT-Tiny semantic segmentation · PASCAL-Context segmentation | OOD Avg. mIoU: 44.00 -> 60.06 | Helped |
| Replace FTP with Adam-SPD | Swin ViT-Tiny semantic segmentation · PASCAL-Context segmentation | OOD Avg. mIoU: 58.45 -> 60.06 | Helped |
Surprising findings
- Uniform L2-SP with strong regularization hurts ID accuracy and fails to improve OOD, while selective regularization improves both.
- Under SPD the correlation between OOD accuracy and deviation on DomainNet is -0.96.
Limitations
- May hurt pre-training because it hinders training of some layers; designed for fine-tuning only.
- Gains depend on how well the foundation model saw the fine-tuning/OOD distributions (e.g., Quickdraw in DomainNet).
- Implementation approximates the original L2-SP update to save memory, slightly dampening the learning rate.
Experimental setup
Datasets: DomainNet ImageNet PASCAL-Context Commonsense-170K VQAv2
Benchmarks: DomainNet ImageNet-V2 ImageNet-A ImageNet-R ImageNet-Sketch PASCAL-Context corruptions BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e
Metrics: ID accuracy OOD accuracy mIoU deviation from initialization
Baselines: AdamW L2-SP MARS-SP TPGM FTP WiSE-FT LP-FT SAM LLRD Adapter BitFit
Builds on: L2-SP Adam LoRA hyper-optimization WiSE-FT
Closest papers by meaning
- Why Do We Need Weight Decay in Modern Deep Learning? (NeurIPS 2024 · similarity 0.89)
- Directional Gradient Projection for Robust Fine-Tuning of Foundation Models (ICLR 2025 · similarity 0.88)
- Implicit Bias of AdamW: $\ell_\infty$-Norm Constrained Optimization (ICML 2024 · similarity 0.87)
- ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflections (ICML 2024 · similarity 0.87)
- Scaling Laws for Sparsely-Connected Foundation Models (ICLR 2024 · similarity 0.86)
- AdaLRS: Loss-Guided Adaptive Learning Rate Search for Efficient Foundation Model Pretraining (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.