Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes dual risk minimization (DRM), combining ERM on default prompts with a worst-case-risk proxy that distills pre-trained CLIP's concept-description predictions, to improve OOD robustness when fine-tuning zero-shot CLIP models.
Problem
Fine-tuning zero-shot foundation models such as CLIP loses robustness to distribution shifts; existing robust fine-tuning methods preserve pre-trained features indiscriminately, without distinguishing robust core features from non-core ones.
Approach
Combine ERM (default prompts, one-hot/contrastive loss) with a worst-case risk proxy (cross-entropy against core-feature soft labels from LLM-generated concept descriptions scored by the pre-trained CLIP with min-max affinity normalization), and infer with a mixture of default and concept-description classifiers.
Key results
- On CLIP ViT-L/14@336, DRM+WiSE-FT reaches 77.1 OOD on ImageNet (vs 75.9 FLYP+WiSE-FT), 51.8 on iWildCam (vs 47.1), and 53.1 on FMoW (vs 50.7).
- On CLIP ViT-B/16 without WiSE-FT, DRM improves iWildCam OOD from 35.6 (FLYP) to 40.0 and FMoW OOD from 41.3 to 45.9.
- Using Llama-3 8B to generate concept descriptions still gives DRM 39.2 OOD on iWildCam vs 35.6 for FLYP (with ViT-B/16 results); DRM is insensitive to LLM choice.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the WRM loss to default-prompt ERM (ablation) | DRM on CLIP ViT-L/14 · iWildCam | macro F1 OOD: 41.9 -> 49.2 | Helped |
| Add the ERM loss to WRM with concept descriptions (ablation) | DRM on CLIP ViT-L/14 · iWildCam | macro F1 ID: 51.7 -> 61.8 | Helped |
| Use default prompts for ERM and concept descriptions for WRM instead of the same prompts for both (ablation) | DRM on CLIP ViT-L/14 · iWildCam | macro F1 OOD: 46.0 -> 49.2 | Helped |
| Use default prompts for ERM and concept descriptions for WRM instead of the same prompts for both (ablation) | DRM on CLIP ViT-L/14 · iWildCam | macro F1 ID: 54.0 -> 61.8 | Helped |
| Use dual-prompt predictions at inference instead of default-prompt predictions only (ablation) | DRM on CLIP ViT-L/14 · iWildCam | macro F1 OOD: 45.1 -> 49.2 | Helped |
| Use dual-prompt predictions at inference instead of concept-description predictions only (ablation) | DRM on CLIP ViT-L/14 · iWildCam | macro F1 ID: 54.8 -> 61.8 | Helped |
| Normalize affinities when estimating core-risk targets (ablation) | DRM on CLIP ViT-L/14 · iWildCam | macro F1 OOD: 24.2 -> 49.2 | Helped |
| Increase λ from 0.0 to 3.0 (ablation) | DRM on CLIP ViT-L/14 · iWildCam | macro F1 OOD: 41.9 -> 49.2 | Helped |
| Increase λ from 0.0 to 3.0 (ablation) | DRM on CLIP ViT-B/16 · ImageNet | OOD accuracy: 60.2 -> 63.2 | Helped |
| Fine-tune with DRM instead of FLYP | CLIP ViT-B/16 · ImageNet OOD average | accuracy: 60.2 -> 63.2 | Helped |
| Fine-tune with DRM instead of FLYP | CLIP ViT-B/16 · iWildCam OOD | macro F1: 35.6 -> 40.0 | Helped |
| Fine-tune with DRM instead of FLYP | CLIP ViT-B/16 · FMoW worst-region OOD | accuracy: 41.3 -> 45.9 | Helped |
Surprising findings
- Using concept descriptions for the ERM term as well degrades performance, so the two terms use different prompt sets.
- Direct (unnormalized) affinity estimates of core-feature probabilities severely degrade both ID and OOD performance (iWildCam OOD 24.2).
Limitations
- Relies on LLM-generated concept descriptions whose quality varies across LLMs; smaller LLMs give weaker gains.
- Requires class labels to compute the core-feature target, so it cannot be used at inference for that component and is limited to labeled training data.
- Theoretical justification relies on approximating worst-case risk with a single-domain proxy and assumptions about diverse domains.
Experimental setup
Datasets: ImageNet WILDS-iWildCam WILDS-FMoW Dollar Street-DA GeoYFCC-DA Hard ImageNet
Benchmarks: ImageNet ImageNet-V2 ImageNet-R ImageNet-Sketch ImageNet-A ObjectNet WILDS-iWildCam WILDS-FMoW Dollar Street-DA GeoYFCC-DA
Metrics: accuracy macro F1 worst-region accuracy
Baselines: FLYP LP FT L2-SP LP-FT WiSE-FT CLIPood TPGM LipSum-FT CaRot
Builds on: FLYP WiSE-FT LP-FT GroupDRO CLIP
Closest papers by meaning
- Robust Fine-tuning of Zero-shot Models via Variance Reduction (NeurIPS 2024 · similarity 0.88)
- LR0.FM: Low-Res Benchmark and Improving robustness for Zero-Shot Classification in Foundation Models (ICLR 2025 · similarity 0.86)
- Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning (CVPR 2024 · similarity 0.86)
- Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidance (ICLR 2024 · similarity 0.86)
- Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws (ICML 2025 · similarity 0.85)
- Foundation Model-oriented Robustness: Robust Image Model Evaluation with Pretrained Models (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.