Research map / Architectures and efficiency / Pruning and efficient tuning
Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, Percy Liang
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use linear-probed head initialization before full fine-tuning (LP-FT) instead of FT | pretrained models across the distribution-shift benchmarks · ID average | accuracy: 85.1 -> 85.7 | Improved |
| Use LP-FT instead of FT | pretrained models across the distribution-shift benchmarks · OOD average | accuracy: 59.3 -> 68.9 | Improved |
| Freeze pretrained features and train only the head (LP) instead of full fine-tuning (FT) | pretrained models with good features and substantial distribution shift · OOD average | accuracy: 59.3 -> 66.2 | Improved |
| Use FT instead of LP | pretrained models across the distribution-shift benchmarks · ID average | accuracy: 82.9 -> 85.1 | Improved |
| Use LP-FT instead of LP | CIFAR-10.1 transfer · CIFAR-10.1 OOD | accuracy: 82.7 -> 93.5 | Improved |
| Select the FT checkpoint and learning rate using OOD accuracy (rather than ID validation) (ablation) | fine-tuning · OOD average | accuracy: 61.3 vs. 67.1 | Hurt |
| Initialize features from the worse MoCo-v1 checkpoint rather than MoCo-v2 (ablation) | Living-17 transfer · Living-17 OOD | accuracy: 71 vs. 66 | Hurt |
| Use FT rather than LP when pretrained features are worse (MoCo-v1) (ablation) | Living-17 transfer with MoCo-v1 features · Living-17 OOD | accuracy: 66 -> 71 | Improved |
| Use FT rather than LP when ID and OOD are very similar (ablation) | CIFAR-10 to CIFAR-10.1 transfer · CIFAR-10.1 OOD | accuracy: 82.7 -> 92.3 | Improved |
| Use LP-FT rather than FT (ablation) | CIFAR-10 to CIFAR-10.1 transfer · CIFAR-10.1 OOD | accuracy: 92.3 -> 93.5 | Improved |
| Fine-tune pretrained features rather than keeping them frozen (ablation) | Living-17 examples · ID versus OOD feature displacement | Euclidean feature distance: 0.0188 ± 0.0001 vs. 0.0167 ± 0.0001 | Mixed |
| Use LP-FT rather than FT (ablation) | Living-17 examples · ID feature displacement | Euclidean feature distance: 0.0188 ± 0.0001 -> 0.0011 ± 0.0001 | Improved |
Closest papers by meaning
- Fine-Tuning is Fine, if Calibrated (NeurIPS 2024 · similarity 0.88)
- Domain-Aware Fine-Tuning: Enhancing Neural Network Adaptability (AAAI 2024 · similarity 0.88)
- LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views (ICML 2024 · similarity 0.87)
- Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods (ICML 2025 · similarity 0.86)
- TTE: Two Tokens Are Enough to Improve Parameter-Efficient Tuning (AAAI 2025 · similarity 0.86)
- CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
