Research map / Architectures and efficiency / Pruning and efficient tuning
How to Fine-Tune Vision Models with SGD
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows AdamW beats SGD when fine-tuning ViTs and ConvNeXts on distribution-shift benchmarks, traces the gap to large first-layer (embedding) gradients, and proposes freezing the embedding layer so SGD matches or beats AdamW with less memory.
Problem
SGD is cheaper in memory than AdamW for fine-tuning large vision models, but it often underperforms AdamW, especially out-of-distribution, on modern ViT and ConvNeXt models.
Approach
Large gradients in the first embedding layer cause SGD to over-tune it; freezing the embedding layer (under 1% of parameters) lets SGD (with or without momentum) match or beat AdamW OOD while using less memory.
Key results
- SGD (freeze-embed, no momentum) reaches 76.9% average OOD accuracy vs 71.9% for SGD and 76.0% for AdamW across 7 models and 5 datasets.
- On CLIP ViT-L/14, SGD (freeze-embed) reaches 83.0% average OOD vs 66.8% for SGD and 82.2% for AdamW; state-of-the-art OOD on Living-17, FMoW and Camelyon.
- On CLIP ViT-L/14, AdamW uses 7.1 GB vs 4.8 GB for SGD (freeze-embed, no momentum) in memory profiling on a Titan-X.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Freeze the embedding layer during SGD fine-tuning | SGD on seven pretrained models · Five distribution-shift benchmarks, average OOD | accuracy: 71.9% -> 76.7% | Helped |
| Remove momentum while freezing the embedding layer | SGD (freeze-embed) · Five distribution-shift benchmarks, average OOD | accuracy: 76.7% -> 76.9% | Helped |
| Freeze the embedding layer during SGD fine-tuning | SGD on seven pretrained models · Five distribution-shift benchmarks, average ID | accuracy: 90.2% -> 91.3% | Helped |
| Replace SGD with AdamW | Fine-tuning seven pretrained models · Five distribution-shift benchmarks, average OOD | accuracy: 72.0% -> 76.0% | Helped |
| Do not freeze the embedding layer's following layer norm (ablation) | SGD (freeze-embed) on CLIP ViT-B/16 · Five benchmarks, average OOD | accuracy: 75.9 -> 75.9 | No clear effect |
| Use 5x lower learning rate on the embedding layer instead of freezing it (ablation) | SGD on CLIP ViT-B/16 · Five benchmarks, average OOD | accuracy: 67.9 -> 75.0 | Helped |
| Remove momentum without freezing the embedding layer (ablation) | SGD on CLIP ViT-B/16 · Five benchmarks, average OOD | accuracy: 67.9 -> 67.9 | No clear effect |
| Add weight decay to SGD (ablation) | SGD on CLIP ViT-B/16 · Five benchmarks, average OOD | accuracy: 67.9 -> 67.8 | No clear effect |
| Freeze only the stem rather than the stem and first stage (ablation) | SGD fine-tuning ConvNeXt-Base · Five benchmarks, average OOD | accuracy: 79.4 -> 78.7 | Hurt |
| Freeze the embedding layer during AdamW fine-tuning (ablation) | AdamW on seven pretrained models · Five distribution-shift benchmarks, average OOD | accuracy: 76.0% -> 76.5% | No clear effect |
Surprising findings
- Freezing only the embedding layer (under 1% of parameters) closes most of the SGD-AdamW gap, while AdamW (freeze-embed) gives no complementary gain, suggesting the two methods help for the same reason.
- Freezing the embedding layer is an ad hoc tweak; linear probing as an alternative freezing scheme performs much worse OOD on FMoW (29.0% vs 37.3% for SGD on CLIP ViT-B/16).
Limitations
- Gains are comparable rather than clearly superior to AdamW in accuracy.
- Freeze choice is architecture-specific and heuristic.
- Memory profiling is limited to one GPU and micro-batch size 1.
Experimental setup
Datasets: Living-17 Waterbirds DomainNet WILDS-FMoW WILDS-Camelyon CIFAR-10
Benchmarks: Living-17 Waterbirds DomainNet WILDS-FMoW WILDS-Camelyon CIFAR-10
Metrics: in-distribution accuracy out-of-distribution accuracy GPU memory (GB)
Baselines: SGD AdamW LARS LAMB LP-FT Gradual-unfreezing Linear probing
Builds on: AdamW LARS LAMB LP-FT
Closest papers by meaning
- PROFIT: A Specialized Optimizer for Deep Fine Tuning (NeurIPS 2025 · similarity 0.85)
- Addax: Utilizing Zeroth-Order Gradients to Improve Memory Efficiency and Performance of SGD for Fine-Tuning Language Models (ICLR 2025 · similarity 0.85)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.84)
- On the $O(\frac{\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\ell_1$ Norm (NeurIPS 2025 · similarity 0.84)
- Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning (ICML 2024 · similarity 0.84)
- SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.