Research map / Vision-language and multimodal / CLIP and zero-shot
Efficient and Versatile Robust Fine-Tuning of Zero-shot Models
Sungyeon Kim, Boseung Jeong, Donghyun Kim, Suha Kwak
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add adapter dropping to R-Adapter (ablation) | R-Adapter · ImageNet ViT-B/32 | OOD avg accuracy: 47.7 -> 49.6 | Improved |
| add re-scaling (weight-space ensemble) to R-Adapter (ablation) | R-Adapter · ImageNet ViT-B/32 | OOD avg accuracy: 47.7 -> 53.5 (ID 77.5 -> 76.5) | Improved |
| add accumulation (EMA of adapter weights) (ablation) | R-Adapter · ImageNet ViT-B/32 | OOD avg accuracy: 47.7 -> 47.8 | No clear effect |
| replace dropout or drop-path with adapter dropping (ablation) | R-Adapter · ImageNet ViT-B/32 | OOD avg accuracy: DO 48.7, DP 47.9, AD 49.6 | Improved |
| replace InfoNCE with MPM-NCE (ablation) | R-Adapter with AD+AC+RS · ImageNet ViT-B/32 | ID and OOD avg accuracy: ID 76.6 -> 77.5, OOD 53.7 -> 53.9 | Improved |
| add label smoothing to InfoNCE (ablation) | R-Adapter · ImageNet ViT-B/32 | ID accuracy: 76.6 -> 76.9 ID is E7 vs E8; OOD 53.7 -> 54.0 | Hurt |
| add label smoothing to MPM-NCE (ablation) | R-Adapter · ImageNet ViT-B/32 | ID and OOD avg accuracy: 77.5 -> 77.7 ID, 53.9 -> 54.3 OOD | Improved |
| use margin with single positive (no soft labels) (ablation) | MPM-NCE · ImageNet ViT-B/32 | OOD avg accuracy: 54.3 -> 47.0 | Hurt |
| increase adapter rank from 4 to full (ablation) | R-Adapter · ImageNet ViT-B/32 | ID / OOD avg accuracy: ID 72.5 -> 77.7, OOD 51.7 -> 54.3 | Improved |
| use adapter drop probability p=0.2 versus 0 (ablation) | R-Adapter · ImageNet ViT-B/32 | OOD avg accuracy: 53.3 -> 54.3 | Improved |
| use R-Adapter instead of full fine-tuning or robust baselines with far fewer trainable parameters | CLIP ViT-B/16 · ImageNet OOD avg | top-1 accuracy: Mask-Fill 63.3 -> 64.8 | Improved |
| use R-Adapter for open-vocab segmentation instead of OVSeg two-stage fine-tuning | OVSeg · A-150 | mIoU: 29.6 -> 29.5 | No clear effect |
Closest papers by meaning
- Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidance (ICLR 2024 · similarity 0.88)
- Zero-Shot Robustification of Zero-Shot Models (ICLR 2024 · similarity 0.88)
- Towards Calibrated Robust Fine-Tuning of Vision-Language Models (NeurIPS 2024 · similarity 0.86)
- Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models (NeurIPS 2024 · similarity 0.86)
- Conformal Prediction for Zero-Shot Models (CVPR 2025 · similarity 0.86)
- Robust Fine-tuning of Zero-shot Models via Variance Reduction (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
