Research map / Label-efficient and robust learning / OOD and anomaly detection
Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that long prompt-learning finetuning of CLIP overfits known classes, and proposes OGEN: a class-conditional feature generator that synthesizes OOD image features from unknown class names plus adaptive local mean-teacher self-distillation to reduce overfitting.
Problem
Finetuning vision-language models such as CLIP with prompt learning improves ID accuracy but, with long training and no proper regularization, overfits known classes and degrades accuracy on unknown (OOD) classes under base-to-new and cross-dataset settings.
Approach
Synthesize OOD image features conditioned on the text feature of an unknown class name by extrapolating from kNN known classes (one lightweight MHCA layer), jointly train known and synthesized unknown features, and regularize the generator with an adaptive local mean teacher whose EMA window grows along a cosine schedule.
Key results
- On base-to-new over 11 datasets, OGEN raises CoOp's new-class accuracy from 63.22% to 69.54% and H from 71.66 to 75.87.
- OGEN-CoOp improves average cross-dataset accuracy from 63.88% to 65.76% when trained on ImageNet and tested on 10 targets.
- OGEN improves new-class accuracy for all six prompt-learning baselines on average (e.g., PromptSRC 76.10 to 76.86, SHIP 73.69 to 76.14).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add OGEN (feature generator + ALMT) to CoOp | CoOp · Base-to-new, avg 11 datasets | New accuracy: +6.32 | Helped |
| add OGEN to CoOp | CoOp · Base-to-new, avg 11 datasets | Base accuracy: +0.78 | Helped |
| add OGEN to CoCoOp, VPT, SHIP, KgCoOp, MaPLe, PromptSRC | prompt learning baselines · Base-to-new, avg 11 datasets | New accuracy: +0.76 to +2.45 for CoCoOp/VPT/SHIP/KgCoOp/MaPLe/PromptSRC | Helped |
| add OGEN to CoOp in cross-dataset setting (ImageNet source) | CoOp · Cross-dataset, 10 targets average | accuracy: +1.88 | Helped |
| add OGEN to CoCoOp in cross-dataset setting | CoCoOp · Cross-dataset, 10 targets average | accuracy: +0.79 | Helped |
| add class-conditional feature generator (no self-distillation) (ablation) | CoOp · Base-to-new ablation, 11 datasets | New accuracy: 63.22 -> 69.02 | Helped |
| add ALMT self-distillation on top of feature generator (ablation) | CoOp + feature generator · Base-to-new ablation, 11 datasets | H: 75.15 -> 75.88 | Helped |
| extrapolate jointly vs extrapolate per class (ablation) | feature generator · Base-to-new ablation, 11 datasets | New accuracy: 66.04 -> 69.02 | Helped |
| extrapolate from kNN known classes vs no extrapolation (ablation) | feature generator · Base-to-new ablation, 11 datasets | New accuracy: 64.08 -> 69.02 | Helped |
| kNN retrieval of base classes vs random base classes (K=3) (ablation) | feature generator · Base-to-new ablation, 11 datasets | H: 75.15 -> 74.40 | Helped |
| Mean Teacher self-distillation instead of none (ablation) | feature generator · Base-to-new ablation, 11 datasets | H: 75.15 -> 75.08 | No clear effect |
| adaptive local window (ALMT m_t) vs fixed window size m=2 or m=9 (ablation) | self-distillation · Base-to-new ablation, 11 datasets | H: 74.50/74.77 -> 75.88 | Helped |
Surprising findings
- Plain Mean Teacher self-distillation does not help (H 75.08 vs 75.15 without distillation), while the local adaptive variant is needed for gains.
- Feature synthesis was reported to be about 3.5x and 7.5x more data-efficient than replaying real LAION-400M OOD data with hard mining and random sampling respectively.
Limitations
- Gains on CoCoOp and VPT are limited because their short training schedules induce little overfitting.
- Only prompt learning is studied; adaptor tuning and other finetuning methods are left to future work.
- Cross-dataset gains are modest on some datasets, and base-class accuracy drops slightly on some datasets.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVC-Aircraft SUN397 UCF101 DTD EuroSAT LAION-400M
Benchmarks: Base-to-new class generalization on 11 datasets Cross-dataset generalization from ImageNet to 10 targets
Metrics: base accuracy new accuracy harmonic mean (H) cross-dataset accuracy
Baselines: CoOp CoCoOp VPT SHIP KgCoOp MaPLe PromptSRC Mean Teacher
Builds on: CoOp CLIP Mean Teacher VOS KgCoOp
Closest papers by meaning
- CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection (ICML 2024 · similarity 0.92)
- Anchor-based Robust Finetuning of Vision-Language Models (CVPR 2024 · similarity 0.91)
- Reproducible Vision-Language Models Meet Concepts Out of Pre-Training (CVPR 2025 · similarity 0.90)
- Towards Calibrated Robust Fine-Tuning of Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- Weak Distribution Detectors Lead to Stronger Generalizability of Vision-Language Prompt Tuning (AAAI 2024 · similarity 0.89)
- Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.