AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Full Fine-Tuning Is Not the Problem: Why Adam Fails and SGD Succeeds in Few-Shot CLIP Adaptation

Nairouz Mrabah, Ismail Ayed, Eric Granger

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that dense full fine-tuning of CLIP with plain SGD beats representative PEFT methods in few-shot adaptation, while Adam is what fails, arguing the problem is optimization rather than parameterization.

Problem

Few-shot CLIP adaptation is usually framed as a parameterization issue, with full fine-tuning assumed to overfit and PEFT preferred; the paper questions whether this conclusion conflates parameterization with optimization.

Approach

Attributes the failure of full fine-tuning to the Adam optimizer rather than to the parameterization, showing that dense full fine-tuning with plain SGD outperforms PEFT methods while Adam does not.

Key results

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use plain SGD instead of Adam for dense full fine-tuningCLIP few-shot adaptation · 11 datasets, 3 backbones, 5 shot levelsaccuracy: SGD consistently outperforms Adam at every shot level and backboneImproved
use dense SGD full fine-tuning instead of PEFT (LoRA, CoOp, LP++, SAFT)CLIP few-shot adaptation · 11 datasets, 3 backbonesaccuracy: SGD above PEFT at all shot levels (numbers in figures)Improved
match Adam's first-step update budget to SGD (ablation)Adam, CLIP ViT-B/16 · 6 datasets, 5 shot levelsmean final accuracy (%): 14.51 -> 76.03Improved
match Adam's first-step update budget to SGD (ablation)Adam, SigLIP · 4 datasets, 3 shot levelsmean final accuracy (%): 38.88 -> 76.60Improved
replace Adam with SGD with momentum (ablation)CLIP few-shot · 11 datasetsaccuracy: restores performance to plain SGD levelImproved
add momentum (first moment) on top of plain SGD (ablation)SGD, CLIP few-shot · 11 datasetsaccuracy: SMD matches plain SGDNo clear effect
change Adam beta2 (0.999, 0.99, 0.9) (ablation)Adam, CLIP ViT-B/16 1-shot · DTD, UCF101accuracy: SGD still exceeds all beta2 valuesNo clear effect
use structured preconditioners Shampoo and SOAP instead of SGDCLIP few-shot · few-shot CLIPaccuracy: far below SGDHurt
reduce learning rate or add warmup (ablation)Adam, CLIP few-shot · common six-dataset subset30-epoch accuracy: restores Adam's accuracyImproved
use SGD direction rather than full-train gradient directionCLIP ViT-B/16 on EuroSAT · EuroSAT full-train lossdirectional curvature: 9158 -> 1317Improved
interpolate SGD solutions between different seedsSGD, CLIP few-shot · EuroSAT, Oxford-Pets, DTDloss barrier: barrier zero on interpolation gridImproved

Surprising findings

Experimental setup

Metrics: classification accuracy

Baselines: parameter-efficient fine-tuning methods Adam-based full fine-tuning

Builds on: CLIP

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy