Research map / Architectures and efficiency / Pruning and efficient tuning
Full Fine-Tuning Is Not the Problem: Why Adam Fails and SGD Succeeds in Few-Shot CLIP Adaptation
Nairouz Mrabah, Ismail Ayed, Eric Granger
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that dense full fine-tuning of CLIP with plain SGD beats representative PEFT methods in few-shot adaptation, while Adam is what fails, arguing the problem is optimization rather than parameterization.
Problem
Few-shot CLIP adaptation is usually framed as a parameterization issue, with full fine-tuning assumed to overfit and PEFT preferred; the paper questions whether this conclusion conflates parameterization with optimization.
Approach
Attributes the failure of full fine-tuning to the Adam optimizer rather than to the parameterization, showing that dense full fine-tuning with plain SGD outperforms PEFT methods while Adam does not.
Key results
- Across 11 datasets, 3 CLIP backbones, and 5 shot levels, dense full fine-tuning with plain SGD outperforms representative PEFT methods.
- Adam-based full fine-tuning is reported to fail where SGD succeeds (abstract truncated; exact numbers not stated in provided text).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use plain SGD instead of Adam for dense full fine-tuning | CLIP few-shot adaptation · 11 datasets, 3 backbones, 5 shot levels | accuracy: SGD consistently outperforms Adam at every shot level and backbone | Improved |
| use dense SGD full fine-tuning instead of PEFT (LoRA, CoOp, LP++, SAFT) | CLIP few-shot adaptation · 11 datasets, 3 backbones | accuracy: SGD above PEFT at all shot levels (numbers in figures) | Improved |
| match Adam's first-step update budget to SGD (ablation) | Adam, CLIP ViT-B/16 · 6 datasets, 5 shot levels | mean final accuracy (%): 14.51 -> 76.03 | Improved |
| match Adam's first-step update budget to SGD (ablation) | Adam, SigLIP · 4 datasets, 3 shot levels | mean final accuracy (%): 38.88 -> 76.60 | Improved |
| replace Adam with SGD with momentum (ablation) | CLIP few-shot · 11 datasets | accuracy: restores performance to plain SGD level | Improved |
| add momentum (first moment) on top of plain SGD (ablation) | SGD, CLIP few-shot · 11 datasets | accuracy: SMD matches plain SGD | No clear effect |
| change Adam beta2 (0.999, 0.99, 0.9) (ablation) | Adam, CLIP ViT-B/16 1-shot · DTD, UCF101 | accuracy: SGD still exceeds all beta2 values | No clear effect |
| use structured preconditioners Shampoo and SOAP instead of SGD | CLIP few-shot · few-shot CLIP | accuracy: far below SGD | Hurt |
| reduce learning rate or add warmup (ablation) | Adam, CLIP few-shot · common six-dataset subset | 30-epoch accuracy: restores Adam's accuracy | Improved |
| use SGD direction rather than full-train gradient direction | CLIP ViT-B/16 on EuroSAT · EuroSAT full-train loss | directional curvature: 9158 -> 1317 | Improved |
| interpolate SGD solutions between different seeds | SGD, CLIP few-shot · EuroSAT, Oxford-Pets, DTD | loss barrier: barrier zero on interpolation grid | Improved |
Surprising findings
- Full fine-tuning with plain SGD outperforms PEFT methods in few-shot CLIP adaptation, contrary to the common assumption that full fine-tuning overfits.
Experimental setup
Metrics: classification accuracy
Baselines: parameter-efficient fine-tuning methods Adam-based full fine-tuning
Builds on: CLIP
Closest papers by meaning
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.88)
- ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflections (ICML 2024 · similarity 0.87)
- Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · similarity 0.87)
- Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed (ICML 2025 · similarity 0.87)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.87)
- Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
