Research map / Label-efficient and robust learning / Recognition and long-tail
Rethinking the Hyperparameters for Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Disable momentum on source-similar targets (ablation) | ImageNet-pretrained ResNet-101 fine-tuning · Dogs, Caltech, Indoor | Top-1 error: 17.20 -> 10.12; 13.42 -> 11.61; 22.79 -> 20.76 | Helped |
| Disable momentum on less-similar targets (ablation) | ImageNet-pretrained ResNet-101 fine-tuning · Cars, Flowers, Aircrafts | Top-1 error: 9.08 -> 10.65; 3.12 -> 5.25; 17.40 -> 20.46 | Hurt |
| Disable momentum (ablation) | ImageNet-pretrained ResNet-101 fine-tuning · Birds | Top-1 error: 18.10 -> 17.21 | No clear effect |
| Use HPO rather than default fine-tuning hyperparameters | Fine-tuning across transfer datasets · Seven image-classification datasets | Top-1 error: Birds 18.10 -> 16.34; Dogs 17.20 -> 9.83; Cars 9.10 -> 7.61; Flowers 3.12 -> 2.91; Aircrafts 17.55 -> 12.33; Caltech 13. | Helped |
| Use L2-SP instead of L2 regularization (ablation) | Fine-tuning · Dogs, Caltech, Indoor | average class error: 16.79 -> 13.86; 14.98 -> 14.45; 23.00 -> 21.77 | Helped |
| Use L2-SP instead of L2 regularization (ablation) | Fine-tuning · Birds, Cars, Flowers, Aircrafts | average class error: 22.51 -> 22.32; 10.10 -> 9.59; 5.70 -> 5.28; 13.03 -> 13.31 | Mixed |
| Choose momentum and effective learning rate based on source-target similarity | Fine-tuning · Transfer-learning datasets | Top-1 error | Helped |
| Use reference-based regularization on dissimilar domains (ablation) | Fine-tuning with L2-SP · Birds, Cars, Flowers, Aircrafts | average class error: 22.51 vs 22.32; 10.10 vs 9.59; 5.70 vs 5.28; 13.03 vs 13.31 | Mixed |
Closest papers by meaning
- Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning (CVPR 2024 · similarity 0.89)
- Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR 2024 · similarity 0.89)
- Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods (ICML 2025 · similarity 0.88)
- PROFIT: A Specialized Optimizer for Deep Fine Tuning (NeurIPS 2025 · similarity 0.87)
- Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition (CVPR 2025 · similarity 0.87)
- Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning? (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.