Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Robust Fine-tuning of Zero-shot Models via Variance Reduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Variance Reduction Fine-tuning (VRF), a sample-wise output-space ensemble of zero-shot and fine-tuned CLIP models whose weights come from k-NN distance to a Zero-Shot Failure set, to get both best ID and OOD accuracy.
Problem
ID-OOD trade-offs persist for ensemble-based robust fine-tuning of zero-shot models like CLIP: ensembles that peak for OOD accuracy lose ID accuracy and vice versa.
Approach
Build a Zero-Shot Failure set of training samples the fine-tuned model gets right but the zero-shot model gets wrong; weight the fine-tuned model per test sample by a sigmoid of its k-NN feature distance to this set, motivated by a variance-reduction argument that the optimal weight is proportional to Acc_ft/Acc_zs.
Key results
- On ImageNet with CLIP ViT-B/16 E2E-FT, VRF reaches 82.3 ID and 61.8 average OOD accuracy versus 82.2 and 60.2 for output-space ensemble (OSE).
- On CIFAR-10 to STL-10 with CLIP ViT-B/16 E2E-FT, VRF improves OOD accuracy to 98.4 from 96.6 for OSE while keeping ID at 98.6.
- Applying VRF to FLYP raises average ImageNet shift accuracy from 62.8 (OSE) to 64.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sample-wise VRF instead of OSE | E2E-FT on CLIP ViT-B/16 · ImageNet five shifts (average) | accuracy: 60.2 -> 61.8 | Helped |
| Use sample-wise VRF instead of WSE | E2E-FT on CLIP ViT-B/16 · ImageNet five shifts (average) | accuracy: 60.6 -> 61.8 | Helped |
| Use sample-wise VRF instead of OSE | LP-FT on CLIP ViT-B/16 · ImageNet five shifts (average) | accuracy: 60.9 -> 62.4 | Helped |
| Use sample-wise VRF instead of WSE | LP-FT on CLIP ViT-B/16 · ImageNet five shifts (average) | accuracy: 61.2 -> 62.4 | Helped |
| Use sample-wise VRF instead of WSE | E2E-FT on CLIP ViT-B/16 · STL-10 | accuracy: 97.8 -> 98.4 | Helped |
| Use the ZSF set rather than the entire training set for distance calculations (ablation) | VRF · ImageNet and distribution shifts | accuracy | Helped |
| Vary k in the k-NN distance calculation (ablation) | VRF · ImageNet | accuracy: less than 0.1% | No clear effect |
| Increase p in the k-NN distance calculation (ablation) | VRF · ImageNet OOD | accuracy: smaller than 0.2% | Hurt |
| Use binary distance-based weights instead of sigmoid weights (ablation) | VRF · ImageNet | accuracy: 82.3 -> 81.3 | Hurt |
| Use linear weights instead of sigmoid weights (ablation) | VRF · ImageNet OOD (average shifts) | accuracy: 61.7 -> 61.8 | No clear effect |
| Increase b across different values of a (ablation) | VRF weight function · ImageNet OOD | accuracy | Helped |
| Apply VRF to FLYP instead of using FLYP alone | FLYP on CLIP ViT-B/16 · ImageNet five shifts (average) | accuracy: 60.2 -> 64.0 | Helped |
Surprising findings
- Using the full training set instead of the zero-shot failure set gives a non-monotonic accuracy ratio curve, so the failure set is essential.
- Selective prediction with traditional OOD detectors falls well short of VRF on ImageNet OOD accuracy.
Limitations
- Assumes zero-shot model has useful predictive power on the downstream task; may fail when pre-training knowledge differs greatly.
- Doubles inference and storage cost because both zero-shot and fine-tuned models must run.
- Theory assumes independent residual errors, with only an empirical check for correlated errors.
Experimental setup
Datasets: ImageNet CIFAR-10 Entity-30
Benchmarks: ImageNet ImageNet-V2 ImageNet-Sketch ImageNet-A ImageNet-R ObjectNet CIFAR-10 to STL-10 Entity-30
Metrics: accuracy
Baselines: Zero-shot CLIP Linear classifier E2E-FT LP-FT OSE WSE FLYP MSP Energy MD kNN RMD
Builds on: CLIP WiSE-FT LP-FT FLYP
Closest papers by meaning
- Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models (NeurIPS 2024 · similarity 0.88)
- Towards Calibrated Robust Fine-Tuning of Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidance (ICLR 2024 · similarity 0.85)
- Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness (CVPR 2024 · similarity 0.85)
- Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models (ICML 2024 · similarity 0.84)
- Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.