AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution

Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, Percy Liang

ICLR 2022 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning · 158 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use linear-probed head initialization before full fine-tuning (LP-FT) instead of FTpretrained models across the distribution-shift benchmarks · ID averageaccuracy: 85.1 -> 85.7Improved
Use LP-FT instead of FTpretrained models across the distribution-shift benchmarks · OOD averageaccuracy: 59.3 -> 68.9Improved
Freeze pretrained features and train only the head (LP) instead of full fine-tuning (FT)pretrained models with good features and substantial distribution shift · OOD averageaccuracy: 59.3 -> 66.2Improved
Use FT instead of LPpretrained models across the distribution-shift benchmarks · ID averageaccuracy: 82.9 -> 85.1Improved
Use LP-FT instead of LPCIFAR-10.1 transfer · CIFAR-10.1 OODaccuracy: 82.7 -> 93.5Improved
Select the FT checkpoint and learning rate using OOD accuracy (rather than ID validation) (ablation)fine-tuning · OOD averageaccuracy: 61.3 vs. 67.1Hurt
Initialize features from the worse MoCo-v1 checkpoint rather than MoCo-v2 (ablation)Living-17 transfer · Living-17 OODaccuracy: 71 vs. 66Hurt
Use FT rather than LP when pretrained features are worse (MoCo-v1) (ablation)Living-17 transfer with MoCo-v1 features · Living-17 OODaccuracy: 66 -> 71Improved
Use FT rather than LP when ID and OOD are very similar (ablation)CIFAR-10 to CIFAR-10.1 transfer · CIFAR-10.1 OODaccuracy: 82.7 -> 92.3Improved
Use LP-FT rather than FT (ablation)CIFAR-10 to CIFAR-10.1 transfer · CIFAR-10.1 OODaccuracy: 92.3 -> 93.5Improved
Fine-tune pretrained features rather than keeping them frozen (ablation)Living-17 examples · ID versus OOD feature displacementEuclidean feature distance: 0.0188 ± 0.0001 vs. 0.0167 ± 0.0001Mixed
Use LP-FT rather than FT (ablation)Living-17 examples · ID feature displacementEuclidean feature distance: 0.0188 ± 0.0001 -> 0.0011 ± 0.0001Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy