Research map / Architectures and efficiency / Pruning and efficient tuning
On the Robustness Tradeoff in Fine-Tuning
Kunyang Li, Jean-Charles Noirot Ferrand, Ryan Sheatsley, Blaine Hoak, Yohan Beugin, Eric Pauley, Patrick McDaniel
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune longer after early steps | ViT-B fine-tuning · Caltech256 | adversarial robustness: 25% -> 10% | Hurt |
| fine-tune the first steps | ViT-B fine-tuning · Caltech256 | accuracy and robustness: accuracy ≈ 90% within 1,000 steps | Improved |
| use BitFit instead of average method | ViT-B fine-tuning · CIFAR-10 | Pareto AUC: 75% above average | Improved |
| use BitFit instead of average method | ViT-B fine-tuning · CIFAR-100 | Pareto AUC: 81.5% | Improved |
| use Compacter instead of average method | ViT-B fine-tuning · Caltech256 | Pareto AUC: 57.5% above average | Improved |
| use Compacter instead of average method | ViT-B fine-tuning · CUB-200 | Pareto AUC: 34.6% above average | Improved |
| use Compacter instead of average method | ViT-B fine-tuning · Stanford Dogs | Pareto AUC: 24% above average | Improved |
| use linear probing rather than other methods | ViT-B fine-tuning · CIFAR-100 | Pareto AUC: 47.5% lower than average | Hurt |
| use full fine-tuning rather than PEFT | ViT-B fine-tuning · CIFAR-100 | Pareto AUC: 20% lower than average | Hurt |
| use full fine-tuning rather than linear probing | ViT-B fine-tuning · DomainNet OOD | OOD robustness: 61% -> 73% | Improved |
| train on a more shifted domain instead of real | ViT-B fine-tuning · DomainNet OOD | OOD robustness: 64% -> 73% | Improved |
| vary LoRA rank (1-20), adapter reduction factor (4-32) or update location (ablation) | PEFT · Adversarial robustness | adversarial robustness | No clear effect |
Closest papers by meaning
- Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks (ICLR 2024 · similarity 0.87)
- Fine-Tuning is Fine, if Calibrated (NeurIPS 2024 · similarity 0.87)
- Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning (CVPR 2024 · similarity 0.87)
- Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning (ICML 2025 · similarity 0.86)
- TuCo: Measuring the Contribution of Fine-Tuning to Individual Responses of LLMs (ICML 2025 · similarity 0.86)
- LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
