Research map / Label-efficient and robust learning / Recognition and long-tail
Fine-Tuning is Fine, if Calibrated
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that fine-tuning a pre-trained classifier on a subset of classes does not damage absent-class features and mainly harms absent-class accuracy via biased logits, so a simple logit calibration factor restores absent-class performance.
Problem
Full fine-tuning on a subset of classes drastically degrades accuracy on the absent classes the pre-trained model had learned (holistic transfer), and it is unclear what component of the model is damaged.
Approach
Dissects the fine-tuned classifier into feature extractor and classifier components, finding features improve for absent classes and the damage is biased logits toward fine-tuning classes; a post-processing additive logit bias on absent classes (with ALG or PCV estimation of gamma) recovers absent-class accuracy.
Key results
- On ImageNet-R/S, naive fine-tuning drops absent-class accuracy AccU/Y from 23.8% to 3.5% while fine-tuning + calibration (gamma by PCV) raises it to 54.0%, outperforming Tu et al. 2023 (23.1%).
- NCM accuracy on absent classes improves after fine-tuning on ImageNet-R/S (AccU/U gain of 21.6%), indicating feature extractor improvement rather than degradation.
- AUSUC on ImageNet-Variants rises from 0.083 (pre-trained) to 0.439 (fine-tuning) versus 0.317 for Tu et al. 2023.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add post-processing calibration to fine-tuning | Fine-tuned ResNet-50 · ImageNet-{R, S} | AccU/Y: 3.5 -> 30.5 | Helped |
| Add post-processing calibration to fine-tuning | Fine-tuned CLIP ViT-B/32 · VTAB | AccU/Y: 39.1 -> 48.2 | Helped |
| Add post-processing calibration to fine-tuning | Fine-tuned ResNet-50 · Office-Home | AccU/Y: 22.4 -> 44.9 | Helped |
| Add PCV calibration rather than no calibration | Fine-tuned model · ImageNet-{R, S} | AccU/Y: 3.5 -> 54.0 | Helped |
| Add PCV calibration rather than no calibration | Fine-tuned model · VTAB | AccY/Y: 62.8 -> 57.4 | Hurt |
| Add PCV calibration rather than no calibration | Fine-tuned model · Office-Home | AccY/Y: 53.5 -> 72.2 | Helped |
| Freeze the linear classifier during fine-tuning instead of updating it (ablation) | Fine-tuning · ImageNet-Variants | AUSUC: 0.439 -> 0.356 | Hurt |
| Freeze the linear classifier during fine-tuning instead of updating it (ablation) | Fine-tuning · Office-Home | AUSUC: 0.632 -> 0.603 | Hurt |
| Freeze the feature backbone and fine-tune only the FC layer (ablation) | Fine-tuning · ImageNet-Variants | AUSUC: 0.439 -> 0.218 | Hurt |
| Freeze the feature backbone and fine-tune only the FC layer (ablation) | Fine-tuning · Office-Home | AUSUC: 0.632 -> 0.595 | Hurt |
| Use a cosine classifier instead of the standard NN classifier (ablation) | Fine-tuned model · ImageNet-Variants | AccU/Y: 3.53 -> 3.77 | No clear effect |
| Use a cosine classifier instead of the standard NN classifier (ablation) | Fine-tuned model · Office-Home | AccU/Y: 22.37 -> 29.83 | Helped |
Surprising findings
- Fine-tuned feature extractors improve absent-class NCM accuracy instead of degrading them, despite absent-class data never being used in fine-tuning.
- Fine-tuning on a subset of classes with a simple post-hoc logit bias beats the prior SOTA holistic transfer method in many benchmarks without complicated training.
Limitations
- Explanation of feature improvement is preliminary and based on a two-layer linear toy analysis; no full theory.
- Frozen classifier and linear probing are worse; benign behavior is weaker with very small fine-tuning class sizes and with adaptive optimizers under poor hyperparameters.
- Calibration factor gamma still requires estimation; the best-case gamma uses test data, so practical results depend on ALG/PCV estimators.
Experimental setup
Datasets: ImageNet-R ImageNet-S Office-Home VTAB
Benchmarks: ImageNet-R ImageNet-S Office-Home VTAB
Metrics: accuracy (AccS/Y, AccU/Y, AccU/U) AUSUC NCM accuracy
Baselines: Tu et al. 2023 Pre-trained model Weight interpolation Frozen classifier Linear probing Oracle fine-tuning
Builds on: Holistic Transfer (Tu et al. 2023) zero-shot logit adjustment (Chao et al.) Nearest Class Mean
Closest papers by meaning
- Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks (ICLR 2024 · similarity 0.88)
- Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods (ICML 2025 · similarity 0.87)
- Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts (ICML 2024 · similarity 0.87)
- Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting (ICML 2025 · similarity 0.86)
- Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR 2024 · similarity 0.86)
- LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.