Research map / Label-efficient and robust learning / Recognition and long-tail
Surgical Fine-Tuning Improves Adaptation to Distribution Shifts
Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, Chelsea Finn
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select layers using relative gradient norm instead of tuning all parameters | Fine-tuning pretrained models · CIFAR-C | accuracy: 79.9 -> 81.4 | Improved |
| Select layers using relative gradient norm instead of tuning all parameters | Fine-tuning pretrained models · ImageNet-C | accuracy: 50.7 -> 51.2 | Improved |
| Select layers using relative gradient norm instead of tuning all parameters | Fine-tuning pretrained models · Living-17 | accuracy: 92.8 -> 93.5 | Improved |
| Select layers using relative gradient norm instead of tuning all parameters | Fine-tuning pretrained models · Entity-30 | accuracy: 79.3 -> 80.6 | Improved |
| Select layers using relative gradient norm instead of tuning all parameters | Fine-tuning pretrained models · CIFAR-Flip | accuracy: 85.9 -> 87.7 | Improved |
| Select layers using relative gradient norm instead of tuning all parameters | Fine-tuning pretrained models · Waterbirds | accuracy: 88.0 -> 88.0 | No clear effect |
| Select layers using relative gradient norm instead of tuning all parameters | Fine-tuning pretrained models · CelebA | accuracy: 82.2 -> 82.2 | No clear effect |
| Use Auto-SNR layer selection instead of tuning all parameters | Fine-tuning pretrained models · CIFAR-Flip | accuracy: 85.9 -> 17.3 | Hurt |
| Use Auto-RGN layer selection rather than manual cross-validation over blocks | Surgical fine-tuning · CIFAR-C | accuracy: 82.8 -> 81.4 | Hurt |
| Tune early layers rather than all parameters for input-level shifts | Surgical fine-tuning · CIFAR-C and ImageNet-C | accuracy | Improved |
| Tune middle layers rather than all parameters for feature-level shifts | Surgical fine-tuning · Living-17 and Entity-30 | accuracy | Improved |
| Tune the last layer rather than all parameters for output-level shifts | Surgical fine-tuning · CIFAR-Flip, Waterbirds, and CelebA | accuracy | Improved |
Closest papers by meaning
- Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution (ICLR 2022 · similarity 0.87)
- Domain-Aware Fine-Tuning: Enhancing Neural Network Adaptability (AAAI 2024 · similarity 0.86)
- Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks (ICLR 2024 · similarity 0.84)
- Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning (CVPR 2024 · similarity 0.84)
- Neural Fine-Tuning Search for Few-Shot Learning (ICLR 2024 · similarity 0.84)
- Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
